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Highlights 



Chapter 1 

• In July 1999, the English Language Acquisition Program (ELAP) was established 
by Assembly Bill 1 1 16 in order “to improve the English proficiency of California 
pupils in grades 4 through 8 and to better prepare them to meet the state academic 
content and performance standards.” ELAP pursues this goal by providing funds 
to districts to be spent on English Learner instruction. 

• Any school district, county office of education, or charter school that has enrolled 
one or more English learners in grades 4 through 8 in the previous school year is 
eligible to participate, provided they follow certain state guidelines. 

• The objectives of this evaluation are to understand how ELAP is being 
implemented, what effect it is having, and how it can be improved. 

• This report addresses the following research questions from the RFP: 

o How are various provisions of Proposition 227 and ELAP being 
implemented in California schools, districts, and the University of 
California? 

o How have the implementation of Proposition 227 and ELAP provisions 
affected the academic achievement of EL students, as measured by STAR 
results, redesignation rates, drop-out rates, high school graduation exam 
passing rates, and high school graduation rates? 

o What changes would strengthen Proposition 227 and ELAP 
implementation and impact? 

Chapter 2 

• Approximately 70 percent of the districts in California have at least one EL in 
grades 4 through 8, and are therefore eligible for ELAP funds. Over the first four 
years, the percentage of ELs in districts receiving funds increased from 91 to 98 
percent. 

• Across the first four years of implementation, uses of ELAP funds have varied 
widely, from after-school and Saturday programs to staff development to 
newcomer classes. According to the Year 4 ELAP survey, the most commonly 
reported use of funds is for English Language Development (ELD) instruction. 

• When asked to identify the strengths of ELAP, most districts highlighted the 
focus that ELAP funds placed on English learners. 
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Chapter 3 

• The restriction of ELAP funds to grades 4 through 8 was reported as the biggest 
constraint in the Year 4 survey. However, some districts saw this restriction as a 
strength. The second most common constraint cited was uncertainty about 
whether funds would be available. 

• Despite a requirement to do so, only 5 percent of the Year 4 survey respondents 
(26 respondents) indicated that they had conducted a formal evaluation of ELAP. 
Only 7 of the 26 evaluations explicitly discussed the role of ELAP in the district. 

• Five districts responding to the Year 4 survey attached evaluations that reflected a 
clear effort to evaluate the impact of ELAP on student achievement. Their 
analysis plans used measures such as statewide tests to follow the academic 
achievement of ELs or qualitative indicators of student progress. 

• Many districts noted on the Year 4 survey that they were not able to judge the 
impact of ELAP because they do not specifically evaluate ELAP. However, more 
than half the districts reported the belief that ELAP funds had had a moderate to 
large impact on areas of language development, academic achievement on state 
content standards, and school redesignation rates. 

• Attempting to assess the impact of a state program like ELAP based on the 
independent efforts of individual school districts appears unrealistic. 



Chapter 4 

• Multiple regression analyses were used to estimate the relationship between 
ELAP and the academic performance of ELs, while controlling for demographic 
and socioeconomic differences between ELAP and non-ELAP schools. 

• To estimate a possible ELAP “effect” without being able to follow individual 
student achievement over time, three different combinations of EL and RFEP 
student groups were used to assess change in selected pre- and post-ELAP 
measures. These alternative comparisons were analyzed due to state data 
limitations, which precluded use of a single, most preferred analysis. Using only 
EL students in the pre-ELAP measurement and both EL and RFEP students in the 
post-ELAP measurement, a positive and statistically significant relationship was 
found between ELAP and SAT-9 test results across all subjects. The size of this 
relationship is relatively small, with average yearly gains in reading and math of 
approximately 1.4 scaled score points, and a gain in language arts of 0.8 points. 
For the approach comparing ELs only in both pre- and post-ELAP measurements, 
ELAP program participation also shows a modest and statistically significant gain 
in SAT-9 reading and math scores. However, the approach using ELs and RFEP 
students in both pre- and post-ELAP measurements shows no significant 
relationship between ELAP and SAT-9 results. 



ENGLISH LANGUAGE ACQUISITION PROGRAM EVALUATION REPORT 




• Although a causal relationship can not be claimed (i.e., that ELAP funding is the 
cause of observed differences in test scores), the results do suggest a relationship 
that is positive and statistically significant. 

• Another outcome variable of interest, redesignation, is a particularly difficult area 
to explore given the varying criteria for redesignation used throughout the state 
and the many other factors affecting these rates. While more ELs are redesignated 
in ELAP schools versus non-ELAP schools, as expected given the broad program 
participation, the rates of redesignation are higher in non-ELAP schools. 



Chapter 5 

• District self-evaluation is not realistic, as attempting to evaluate the impact of an 
individual funding stream is a complex undertaking. 

• As an alternative, the state should consider possible collaboration with selected 
large districts to enable case study evaluations of ELAP. 

• Some additional useful data regarding the implementation of ELAP could be 
collected for all districts. However, increased capacity of the state and districts to 
administer ELAP may be needed in order to support additional data collection. 

• Use the same test statewide over time to the extent possible, in order to monitor 
the progress of categories of students. Ideally, these longitudinal data would link 
individual students over time. 

• Statewide student outcome data are insufficient to provide clear answers 
regarding the degree of ELAP success, although in this report the research team 
has attempted to address the question of effectiveness to the extent possible given 
these limitations. The analyses in this report generally suggest a small, 
statistically significant relationship between ELAP and EL student academic 
achievement. 

• The research team believes these results, although modest, are sufficiently 
promising to warrant program continuation with ongoing monitoring and 
evaluation. 
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Executive Summary 

Background 

In June of 1998, Proposition 227 was passed by 61 percent of the California electorate. 
Intended to change how the state’s English learners (ELs) are instructed, Proposition 227 
required that ELs be taught “overwhelmingly in English” through sheltered/structured 
English immersion (SEI) programs during a transition period and then transferred to 
English-language mainstream classrooms. Thirteen months later, in July of 1999, the 
English Language Acquisition Program (ELAP) was established by Assembly Bill 1116 
in order “to improve the English proficiency of California pupils in grades 4 through 8 
and to better prepare them to meet the state academic content and performance 
standards.” ELAP pursues this goal by providing funds to districts to be spent on English 
learner instruction. 

This report is part of a five-year study being conducted by the American Institutes for 
Research (AIR) and WestEd for the California Department of Education (CDE). The 
Study, Effects of the Implementation of Proposition 227 on the Education of English 
Learners, 1 evaluates the impact of Proposition 227, and includes an evaluation of ELAP. 
This report presents findings from the ELAP evaluation. 

Objectives of the ELAP Evaluation 

In general, the objectives of this evaluation are to understand how ELAP is being 
implemented, what effect it is having, and how it can be improved. These objectives are 
addressed in the following research questions from the RFP: 

• How are various provisions of Proposition 227 and ELAP being implemented 
in California schools, districts, and the University of California? 

• How have the implementation of Proposition 227 and ELAP provisions 
affected the academic achievement of EL students, as measured by STAR 
results, redesignation rates, drop-out rates, high school graduation exam 
passing rates, and high school graduation rates? 

• What changes would strengthen Proposition 227 and ELAP implementation 
and impact? 

In addition, according to AB 1116, the independent evaluator (i.e., AIR and WestEd) is 
charged with “providing recommendations for modifications to the program,” as well as 
with “comparing the success of participating districts in meeting the goals and objectives 
of the program to non-participating districts, taking into consideration comparisons to 
schools with similar characteristics.” 



1 A full report on the findings and recommendations from the first three years of the study is available 
online at http://www.air.org/publications/publications-set.htm under Elementary and Secondary Education, 
and as www.wested.org/cs/we/view/rs/661 . 
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Evaluation Approach 

This report presents results from a survey conducted by AIR/WestEd and the CDE, 
findings from individual district evaluation efforts, and an analysis of statewide student- 
level data. This report also draws upon data from prior phone interviews, surveys, and 
case study site visits related to the larger Proposition 227 evaluation. The statewide 
analysis examined the relationship between ELAP on EL student achievement, as well as 
redesignation rates over time. This report presents a culmination of findings from all data 
collection efforts over the past four years pertaining to ELAP, including phone 
interviews, two sets of case study site visits, and two sets of survey efforts. 

Survey and Qualitative Findings 

• The Year 4 survey, administered by AIR/WestEd and CDE to districts that had received 
ELAP funds, had a response rate of 95 percent. 

This report draws heavily from data gathered in the Year 4 survey. The survey was sent 
to every district that had received at least one year of ELAP funds, using a list of districts 
provided by the CDE. AIR/WestEd and CDE went to great lengths to ensure that districts 
participated, and out of the 546 districts that received the survey, all but 28 returned it, 
for a response rate of 95 percent. 

• Although less than half of California school districts receive ELAP funds, these 
districts enrolled 98 percent of EL students in grades 4 through 8 in 2002-03. 

403 districts received ELAP funds in the 1999-2000 school year, growing to 516 in the 
2002-03. $51.8 million was allocated in the first year, $70 million in the second, 2 and 
$53.2 million in the two subsequent years. 

Approximately 70 percent of the districts in California have at least one EL in grades 4 
through 8, thereby being eligible for ELAP. Over the first four years of the program, the 
percentage of ELs in districts receiving funds increased from 91 to 98 percent. 

• Although ELAP funds are used for a variety of purposes, the primary use in 2002-03 
was to support ELD instructional programs. 

According to phone interviews, site visits, and our survey efforts, uses of ELAP funds 
have varied widely, including after-school and Saturday programs, staff development, 
and newcomer classes. 

When asked to identify strengths of ELAP in the Year 4 survey, most districts 
highlighted the focus on English Learners as key. When asked about constraints in using 



2 In 2000-01 an extra $16.8 million was allocated to cover a provision of AB 1116 that specified a one-time 
payment of $100 per student for every EL in kindergarten through grade 12 who was “reclassified to 
English-fluent status.” However, the bill required that the CELDT be used to determine English fluency, 
which was not yet available when districts first applied. Thus, this money went unclaimed. 
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the funds, restriction to grades 4 through 8 was viewed as the biggest constraint. The 
second most common constraint cited was uncertainty about future funds. 

Findings from Local Evaluation Efforts 

• Although mandated in AB 1116, districts struggled to perform the required evaluation 
of ELAP, which requires isolating ELAP impact from other EL factors and designing an 
appropriate method for data collection and analysis. This may be beyond the 
evaluative capacity of individual districts. 

Despite a requirement to do so, only 5 percent of the respondents (26 districts) indicated 
that they had conducted a formal evaluation of ELAP. Only 7 of these 26 evaluations 
explicitly discussed the role of ELAP in the district, and only 5 of those 7 submitted 
evaluations that reflected a clear effort to specifically evaluate the impact of ELAP on 
student achievement. Their analysis plans used measures such as statewide tests to follow 
the academic achievement of ELs or qualitative indicators of student progress. This may 
suggest that districts do not know how to isolate the impact of ELAP on ELs, and that 
such evaluations may be beyond the individual capacity of districts, especially if 
examples of evaluative models and methods are not provided. 

Many districts noted that they were unable to judge the impact of ELAP because they do 
not specifically evaluate it. At the same time, more than half the districts reported that 
ELAP funds had a moderate to large impact on areas of language development, academic 
achievement on state content standards, and school redesignation rates. Respondents 
indicated that improved results on English language development assessments were the 
most direct effects of ELAP. 

Findings from Statewide Data 

• ELAP recipient schools are demographically different than non-ELAP recipient 
schools, with higher poverty and higher EL counts. When controlling for these 
differences, overall results suggest that ELAP schools have a small but significant 
increase in reading, math, and language arts achievement. 

Ideally, analyses of ELAP impact would be based on individual student data, linked over 
time. This would have allowed us to track the academic progress of individual students 
over time with varying degrees of exposure to ELAP. However, current state data does 
not allow individual student performance to be followed over multiple years, so instead 
the study used analyses of cohorts of students over time. However, because current 
student-level data do not indicate the year of redesignation for a student, the preferred 
model of cohort tracking was also not possible, i.e. to start with a base of ELs and retain 
them over time, whether redesignated or not. 

As a result, three different approaches were designed to create alternative views of a 
possible ELAP effect. The first approach includes ELs only in the 1998-99 pre-ELAP 
measure, and data for both ELs and RFEPs (ELs redesignated as English-proficient) 
combined for the 2001-02 post-ELAP score. Approach 2 includes ELs in both years, and 
the third approach includes ELs and RFEPs in both the pre- and post-ELAP 
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measurements. ELAP schools were classified in terms of length of exposure to the ELAP 
program. 



The most positive regression results obtained (Approach 1) indicate a yearly positive 
“effect” of about 1.4 scale score points in the areas of math and reading. For language 
arts, a yearly positive “effect” of less than 1 scale score point was revealed, which is 
fairly small. Although we cannot claim a causal relationship from these analyses (i.e. that 
ELAP funding is the cause of the observed rise in test scores), overall, they suggest a 
relationship between ELAP and academic achievement that is positive and statistically 
significant. 

• Redesignation rates are higher in non-ELAP recipient schools, but this may reflect the 
higher poverty and EL counts in ELAP recipient schools. 

Redesignation is a particularly difficult area to explore given the varying criteria for 
redesignation used throughout the state and the many other factors affecting these rates. 
For example, the most noticeable drop in redesignation rates, found in 2000-01, most 
likely results from factors other than ELAP. These other factors include the CELDT, 
which was first administered the prior fall, and the English Language Arts California 
Standards Test (ELA CST), both of which were included in the guidelines adopted by the 
California State Board of Education for redesignation decisions. While more ELs are 
redesignated in ELAP schools versus non-ELAP schools, as expected given the broad 
program participation, the rates of redesignation are higher in non-ELAP schools. This 
may be due to the fact that ELAP-recipient schools have higher concentrations of ELs 
and higher rates of poverty, factors that may negatively influence redesignation. A more 
complete exploration of redesignation rates as an EL outcome variable will be included in 
the final year of this evaluation. 

Conclusion 

ELAP was valued by our survey respondents for targeting funds to the state’s substantial 
population of ELs. Most respondents clearly indicated their belief that the attention raised 
for this at-risk population and the supplemental programs ELAP generated have been 
effective in advancing education outcomes for the state’s ELs. 

At the same time, evidence substantiating their belief in the efficacy of this program has 
generally been lacking. The requirement for meaningful district self-evaluation has not 
been met and was undoubtedly unrealistic. Statewide student outcome data are 
insufficient to provide clear answers in regard the degree of program success. In this 
report, AIR/WestEd has attempted to address the question of ELAP effectiveness to the 
extent possible given these statewide data limitations. The overall analyses contained in 
this report suggest a modest, statistically significant relationship between ELAP and EL 
student achievement. 

AIR/WestEd believes these findings, although modest, are sufficiently compelling to 
warrant program continuation with ongoing monitoring and evaluation. We also suggest 
that program implementation be enhanced to include some of the recommendations 
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included in this report, which would allow better tracking of the extent and ways in which 
the program is impacting the education received by the state’s EL population. 
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Chapter 1. Introduction 



Highlights: 

• In July 1999, the English Language Acquisition Program (ELAP) was established 
by Assembly Bill 1 1 16 in order “to improve the English proficiency of California 
pupils in grades 4 through 8 and to better prepare them to meet the state academic 
content and performance standards.” ELAP pursues this goal by providing funds 
to districts to be spent on English learner instruction. 

• Any school district, county office of education, or charter school that has enrolled 
one or more English learners in grades 4 through 8 in the previous school year is 
eligible to participate, provided they follow certain state guidelines. 

• The objectives of this evaluation are to understand how ELAP is being 
implemented, what effect it is having, and how it can be improved. 

• This report addresses the following research questions from the RFP: 

o How are various provisions of Proposition 227 and ELAP being implemented 
in California schools, districts, and the University of California? 

o How have the implementation of Proposition 227 and ELAP provisions 
affected the academic achievement of EL students, as measured by STAR 
results, redesignation rates, drop-out rates, high school graduation exam 
passing rates, and high school graduation rates? 

o What changes would strengthen Proposition 227 and ELAP implementation 
and impact? 



Overview 

In June of 1998, Proposition 227 was passed by 61 percent of the California electorate. 
Intended to change how the state’s English learners (ELs) are instructed, Proposition 227 
required that ELs be taught “overwhelmingly in English” through sheltered/structured 
English immersion (SEI) programs during a transition period and then transferred to 
English-language mainstream classrooms. Thirteen months later, in July 1999, the 
English Language Acquisition Program (ELAP) was established by Assembly Bill 1116 
in order “to improve the English proficiency of California pupils in grades 4 through 8 
and to better prepare them to meet the state academic content and performance 
standards.” ELAP pursues this goal by providing funds to districts to be spent on English 
learner instruction. 

This report is part of a five-year study being conducted by American Institutes for 
Research (AIR) and WestEd for the California Department of Education (CDE). The 
study, Effects of the Implementation of Proposition 227 on the Education of English 



ENGLISH LANGUAGE ACQUISITION PROGRAM EVALUATION REPORT i-l 




Learners is evaluating the impact of Proposition 227, and includes an evaluation of 
ELAP. This report presents the ELAP evaluation component. The exact provisions of 
ELAP, from California Assembly Bill 1116 of 1999, are shown in Exhibit 1-1 below. 



Exhibit 1-1. ELAP Provisions, from Assembly Bill 1116 



(d) As a condition of receiving funds under subdivision (a), each local educational agency shall certify that it will do all of 
the following: 

(1) Conduct academic assessments of English language learners to ensure appropriate placement of those pupils. The 
assessments shall include: 

(A) Initial assessment of English language learners to determine their English proficiency level. 

(B) Ongoing assessment conducted at least annually to ensure accurate placement of English language learners, to 
communicate progress, and to provide formative assessment information to refine the program. Assessment 
measures shall include, but are not limited to, the state standardized testing and reporting program required by 
Section 60604, unless a pupil is exempted by law, and the English language development instrument to be 
developed pursuant to Section 6081 1 , when it is developed. 

(2) Provide a program for English language development instruction to assist pupils in successfully achieving the 
English language development standards adopted by the State Board of Education pursuant to Section 6081 1 . The 
program shall include structured immersion instruction to be provided for English learners, such as specially designed 
academic instruction in English, and sheltered English strategies to ensure access by English language learners to the 
core curriculum, unless the local educational agency has obtained a waiver pursuant to Section 310. 

(3) Provide supplemental instructional support, such as intersession, before and after school, opportunities or summer 
school to provide English language learners with continuing English language development. These opportunities are to 
supplement the regular school program and may include, but are not limited to, newcomer centers and tutorial support, 
mentors, or any other program that meets the objectives of the program established pursuant to this chapter. Academic 
support services needed to provide these opportunities may be funded by this program. 

(4) Coordinate services and funding sources available to English language learners, including, but not limited to, 
community-based English tutoring programs established pursuant to Article 4 (commencing with Section 315) of 
Chapter 3, programs for at-risk youth, after-school, intersession, and summer school programs, reading programs 
established pursuant to Chapter 2 of the Statutes of 1999 (First Extraordinary Session) and any available federal funds. 
The local educational agency shall also certify that it integrates adult community-based tutoring resources with the 
program established pursuant to this chapter. 



As the legislative provisions above show, ELAP is not, strictly speaking, a program or 
specific type of intervention. Rather, it is a funding source that can be used to support a 
number of possible interventions. Districts receiving ELAP funds can use them very 
flexibly in pursuit of ELAP’s stated goals for ELs in grades 4 through 8. Therefore, we 
are attempting to examine the use of funding for a myriad of possible interventions, and 
attempting to discern the effects of those interventions on the targeted population. Later 
in this report we present information regarding the alternative ways in which ELAP funds 
are being used. 

As examples of ELAP’s flexibility, funds may be used to support possible interventions 
or supplement regular school programs, newcomer centers, tutorial services, mentors, 
purchase of special materials, and other related program activities. Any school district, 
county office of education, or charter school that has enrolled one or more English 
learners in grades 4 through 8 in the previous school year is eligible to participate. As 
noted above, districts must do the following: 



1 A full report on the findings and recommendations from the first two years of the study is available online 
at www.wested.org/cs/we/view/rs/661 or http://www.air.org/publications/publications-set.htm 
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(1) Conduct academic assessments of English learners to ensure their appropriate 
placement. These include initial assessments of English proficiency, and ongoing 
assessments conducted at least annually. 

(2) Provide a program for English language development instruction to assist pupils in 
successfully achieving the English language development standards adopted by the 
State Board of Education. 

(3) Provide supplemental instructional support opportunities for English learners, such 
as intersession, before and after school, or summer school programs. 



ELAP was intended to help English learners meet English language development 
standards, as well as grade-level standards in reading, writing, mathematics, science, and 
history/social science. In addition, the law stipulates that “data developed through this 
program be used to inform curriculum, instruction, assessment, research, inservice staff 
development, and teacher preparation regarding use of the most effective practices for 
teaching English learners.” However, because ELAP is a funding source rather than a 
structured program, and because districts have great flexibility in how they use ELAP 
funds, it is exceedingly difficult to analyze these disparate uses and their potential 
benefits for the targeted population. Further, legislation in the last two years has allowed 
districts to use ELAP funds for purposes unrelated to educating ELs. Also, while funds 
are allocated on a per EL student basis, the total amount of funding available within a 
district or going to a particular school depends on the size of the EL population, so there 
may be an issue of “critical mass” of funding required to implement effective strategies. 
All of these complicating factors will be explored in depth in this report. It is also 
important to note that ELAP did not occur in a vacuum. Other important changes during 
this period include the implementation of Proposition 227, the introduction of the 
California English Language Development Test (CELDT), and the state’s class size 
reduction initiative. The adoption of standards and training associated with each of these 
changes helped shape the context in which ELAP took place. 

Context of the Report 

As described above, this evaluation of ELAP was performed as part of a larger study 
commissioned to answer the eight research questions listed below. The three questions 
relating to the evaluation of ELAP are shown in bold: 

1. How are various provisions of Proposition 227 and ELAP being implemented 
in California schools, districts, and the University of California? 

2. Which programs and services being provided to English learners are most 
effective and least effective in ensuring equal access to the core academic 
curriculum, the achievement of state content and performance standards, and 
rapid acquisition of English? 
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3. What are other program benefits (to parents, teachers, et al.) of the various 
effective programs and services? 

4. What unintended consequences, both positive and negative, have occurred as a 
result of Proposition 227 implementation? 

5. How have the implementation of Proposition 227 and ELAP provisions 
affected the academic achievement of EL students, as measured by STAR 
results, redesignation rates, drop-out rates, high school graduation exam 
passing rates, and high school graduation rates? 

6. What impact have the Professional Development Institutes had on the staff of 
participating ELAP schools? 2 

7. What have been the effects of the CBET programs on the participants and on 
English learner students? 

8. What changes would strengthen Proposition 227 and ELAP implementation 
and impact? 

As shown above, the research questions relevant to ELAP ask how the program is being 
implemented, what effect it is having, and how it can be improved. These questions are 
addressed in this report, using the following techniques: 1) examining the results of a 
survey conducted by AIR/WcstEd and the California Department of Education, 2) 
assessing local evaluation efforts, and 3) analyzing statewide student-level data. The 
report also draws upon data from prior phone interviews, surveys, and case study site 
visits related to the larger Proposition 227 evaluation effort. We describe the 
implementation of ELAP, its issues and challenges, and the evidence of the program’s 
impact. In addition, we consider statewide achievement, fiscal, and demographic data 
relevant to ELAP. We also offer conclusions and recommendations regarding ELAP. 

Evaluating ELAP 

AB 1116 required that districts receiving ELAP funds evaluate how effective the funding 
had been in improving EL instruction and student outcomes. These local evaluation 
reports were mandated to include the following information: 

1. An assessment of the effectiveness of that LEA in helping English language 
learners achieve state academic content and performance standards, including 

• Increasing school rates of redesignation of pupils from English language 
learner to English fluent. 



2 Research question 6 applied to ELAP, but was dropped by agreement between the CDE and project staff 
due to the brief duration of the institutes (funded in 2000-01 and 2001-02, with carryover funds lasting until 
2002-03) and the difficulty of assessing the impact of these programs on staff at ELAP schools. 

3 This report fulfills the requirements of the Final AB 1116 Report, as described in the Request for 
Proposals (RFP). 
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• Increasing high school completion rates of English learners. 

• Improving test scores assessing English language development as well as 
grade level standards in reading, writing, mathematics, science, and 
history/social science. 



2. Problems encountered in the design and operation of the program, including 
identification of any federal, state, or local statutes or regulations that impede the 
program. 



In the survey sent by the CDE and AIR/WestEd, districts were asked to submit a formal 
evaluation of ELAP if one had been conducted. Despite the AB 1116 evaluation mandate, 
of the 518 districts completing the survey, 4 only 26 districts responded with information 
about evaluation efforts they had performed. Of these 26, only 7 provided data explicitly 
related to ELAP. Chapters 2 and 3 address challenges the districts have faced in 
attempting to evaluate the program. The survey collected information on the local 
program’s implementation and effectiveness, and allowed the study team to 
systematically compare information across districts. (See Appendix A for a copy of the 
survey.) Exhibit 1-2 is a crosswalk between the ELAP survey questions and the ELAP- 
related research questions from the request for proposals. Exhibit 1-3 is a crosswalk 
between the survey questions and the legislative requirements of local education agency 
(LEA) ELAP evaluation reports and the legislative requirements of an independent 
evaluator (i.e., AIR/WestEd). 



4 546 districts were sent the CDE/AIR/WestEd surveys. AIR/WestEd received 518 completed surveys, for a 
95 percent response rate. Survey results are discussed in Chapters 2 and 3. 
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Exhibit 1-2. Crosswalk between ELAP Survey Questions and ELAP Research Questions 





Research Questions 






RQ 5. 




ELAP Survey Questions 


RQ 1. 

How are various 
provisions of Proposition 
227 and ELAP being 
implemented in 
California schools, 
districts, and the 
University of California? 


How have the implementation of 
Proposition 227 and ELAP 
provisions affected the academic 
achievement of EL students, as 
measured by STAR results, 
redesignation rates, dropout 
rates, high school graduation 
exam passing rates, and high 
school graduation rates? 


RQ 8. 

What changes would 
strengthen Proposition 
227 and ELAP 
implementation and 
impact? 


1 . In which of the following years did your district receive ELAP 
funding? 


X 


2. Does your district allocate ELAP funds to schools? 


X 


3. What percentage of your total ELAP budget is used centrally by 
the district and what percentage is allocated to schools? 


X 


4. Which of the following best describes the basis upon which 
funds are allocated to schools? (A fixed amount is allocated to 
schools per EL student, a fixed amount is allocated per school or 
type of school, schools receive ELAP funds according to budgets 
that they submit to the district, other) 


X 


5. Which of the following was supported, at least in part, by ELAP 
funds? And approximately what percentage of your district's total 
ELAP budget was allocated to this? (Core academic instructional 
program, ELD instructional program, extended time programs, 
newcomer services, staff development, language testing and 
assessment, etc.) 


X 


6. What percentage of total ELAP funds was used for each of the 
following? (Certificated personnel, non-certificated personnel, 
textbooks/materials/supplies, technology or equipment, other) 


X 


7. To what extent has your district's ability to use ELAP funds been 
constrained by the following? (restriction of funds to grades 4-8, 
uncertainty of available funds, lack of guidance on how funds can 
be used, lack of teachers, lack of classroom space, lack of 
appropriate EL instructional materials, delayed receipt of funds, 
other) 


X 


8. Please describe any other problems your district has 
encountered in the design and operation of ELAP, including 
identification of any federal, state, or local statute that impedes 
program implementation. 


X 


9. Have you used any of the following measures to assess the 
effectiveness of ELAP in improving teaching and learning? 
(CELDT, SAT-9, CA Standards Test, District writing proficiency 
test, other district-wide assessments, non-cognitive indicators, 
teacher surveys, observation of teacher practice, other) 


X 


10. If you answered "yes" to any of the items in Question 9 above, 
please provide a brief description of what you have done in regard 
to each and what you have found. 


X 


1 1 . To what extent has ELAP resulted in each of the following 
outcomes for ELs in your district? (Improved performance on tests 
assessing ELs’ ELD, improved performance on tests assessing 
ELs grade level achievement on state academic content and 
performance standards, increase in school rates of EL 
redesignation to English fluency, increase in ELs high school 
completion rates, other) 


X 


12. Do you have evidence that can be used to substantiate your 
responses to Question 1 1 above? 


X 


13. Please provide a brief narrative presenting the evidence and 
include examples if possible. 


X 


14. Has your district conducted a formal evaluation of ELAP? 




15. Are there any other particular strengths or weaknesses 
associated with ELAP that are important to share? 


X 


X 


X 


16. Is there anything else that you would like to tell us about your 
district's experience with ELAP? 


X 


X 


X 
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Exhibit 1-3. Crosswalk between ELAP Survey Questions and Requirements of Legislation 





Legislative Requirements of LEAs (Per AB 1116 Section 1. Chapter 4. 408 (a)(1) 


Requirements of Independent Evaluator 
Per AB 1116 Section 1 . Chapter 4. 
408 (a)(2)(b) 


ELAP Survey Questions 


A(i). An 
assessment of 
the effectiveness 
in increasing 
school rates of 
redesignation 


A(ii). An 
assessment of 
the effectiveness 
in increasing 
high school 
completion rates 
of ELs 


A(iii). An assessment of 
the effectiveness in 
improving test scores 
assessing ELD, as well as 
grade level standards in 
reading, writing, math, 
science, and history/social 
science 


(B) Locate problems 
encountered in the design 
and operation of the 
program, including 
identification of any federal, 
state, or local statute or 
regulation that impedes 
program implementation 


Recommend- 
ations for 
modifications 
to the program 
to achieve 
goals 


Compare success of 
participating LEAs in 
meeting the goals and 
objectives to LEAs not 
participating in program 
and take into consideration 
comparisons to schools 
with similar characteristics 


1 . In which of the following years did your district 
receive ELAP funding? 






2. Does your district allocate ELAP funds to schools? 






3. What percentage of your total ELAP budget is 
used centrally by the district and what percentage is 
allocated to schools? 






4. Which of the following best describes the basis 
upon which funds are allocated to schools? (A fixed 
amount is allocated to schools per EL student, a 
fixed amount is allocated per school or type of 
school, schools receive ELAP funds according to 
budgets that they submit to the district, other) 






5. Which of the following was supported, at least in 
part, by ELAP funds? And approximately what 
percentage of your district's total ELAP budget was 
allocated to this? (Core academic instructional 
program, ELD instructional program, extended time 
programs, newcomer services, staff development, 
language testing and assessment, etc.) 






6. What percentage of total ELAP funds was used for 
each of the following? (Certificated personnel, non- 
certificated personnel, textbooks/materials/supplies, 
technology or equipment, other) 






7. To what extent has your district's ability to use 
ELAP funds been constrained by the following? 
(restriction of funds to grades 4-8, uncertainty of 
available funds, lack of guidance on how funds can 
be used, lack of teachers, lack of classroom space, 
lack of appropriate EL instructional materials, 
delayed receipt of funds, other) 


X 


X 


8. Please describe any other problems your district 
has encountered in the design and operation of 
ELAP, including identification of any federal, state, or 
local statute that impedes program implementation. 


X 


X 


9. Have you used any of the following measures to 
assess tine effectiveness of ELAP in improving 
teaching and learning? (CELDT, SAT-9, CA 
Standards Test, District writing proficiency test, other 
district-wide assessments, non-cognitive indicators, 
teacher surveys, observation of teacher practice, 
other) 


X 


X 


1 0. If you answered "yes" to any of the items in 
Question 9 above, please provide a brief description 
of what you have done in regard to each and what 
you have found. 


X 


X 


1 1 . To what extent has ELAP resulted in each of the 
following outcomes for ELs in your district? 
(Improved performance on tests assessing ELs’ 

ELD, improved performance on tests assessing ELs 
grade level achievement on state academic content 
and performance standards, increase in school rates 
of EL redesignation to English fluency, increase in 
ELs high school completion rates, other) 


X 


X 


X 




X 


12. Do you have evidence that can be used to 
substantiate your responses to Question 1 1 above? 


X 


X 


X 




X 


13. Please provide a brief narrative presenting the 
evidence and include examples if possible. 


X 


X 


X 




X 


14. Has your district conducted a formal evaluation of 
ELAP? 


X 


X 


X 




X 


15. Are there any other particular strengths or 
weaknesses associated with ELAP that are important 
to share? 


X 


X 


16. Is there anything else that you would like to tell 
us about your district's experience with ELAP? 


X 


X 
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In addition to the survey and the assessment of local evaluation efforts, the research team 
conducted statewide student outcome analyses in an attempt to measure the impact of 
ELAP on EL academic achievement. We also examined redesignation rates over time for 
these students. Although Research Question 5 specifies the use of graduation rates as a 
measure of EL academic achievement for this study, it was inappropriate to analyze high 
school graduation examination passing rates and high school graduation rates given that 
the oldest students in the cohort (i.e., those students who were in 8 th grade during the first 
year of implementation, 1999-2000) are currently in the 12 th grade. 

ELAP Evaluation in Context of the Broader AIR/WestEd Study 

The evaluation of the implementation and effects of ELAP is one of four broad categories 
of EL education issues being examined under the auspices of the larger five-year study. 
The other components under examination include the implementation and effects of 
Proposition 227 on California’s public school system, EL academic achievement, and the 
implementation and potential effects of the Community-Based English Tutoring (CBET) 
Program. This report presents a culmination of findings from all data collection efforts 
over the past four years pertaining to ELAP. During the first three years, the project team 
performed the following research activities: 

• Year 1 activities included phone interviews with district administrators about the 
implementation of Proposition 227, ELAP, and CBET, and case study site visits 
across eight sample districts. These visits included district- and school-level 
interviews, focus groups, and classroom observations about the implementation of 
Proposition 227, ELAP, and CBET (see the Year 1 Final Report for further 
details). 

• Year 2 activities included the development, administration, and analysis of 
surveys to administrators and teachers (which provided district- and school-level 
information on ELAP); analysis of state demographic, instructional, and 
individual student achievement data from 1998 to 2000; and structured interviews 
with 15 key stakeholders who played prominent roles either in supporting or 
opposing Proposition 227 (see the Year 2 Final Report for further details). 

• Year 3 included exploration of the elements of effective practice with ELs, 
selecting a sample of “effective” and comparison schools from across the state to 
explore the elements of effectiveness in case study site visits. Site visits included 
interviews with ELAP coordinators and other district and school personnel 
familiar with the program. Year 3 also included analyses of 1998-2002 statewide 
EL outcome data. (See the Year 3 Final Report for further details.) 

Contents of this Report 

The subsequent chapters of this report are organized as follows. Chapter 2 discusses the 
implementation of ELAP. We describe the system used by the CDE to determine which 
districts receive ELAP funds, and we detail the application process. The chapter presents 
ELAP funding patterns and presents demographic information about the districts that 
receive the funds. The chapter concludes with a discussion of the use of funds within the 
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districts and schools, and of the issues and challenges faced in using the funds. Chapter 3 
presents qualitative data regarding the program. These data are primarily drawn from a 
district-level survey of all ELAP recipients. The chapter also draws upon prior data 
collected throughout this study. These include Year 1 district phone interviews, Year 2 
and 4 surveys, and case study site visit data from Years 1 and 3. The chapter also 
includes a description of the formal ELAP evaluations that have been conducted in 
various districts. Chapter 4 analyzes ELAP’s potential impact based on statewide 
achievement and redesignation data. Chapter 5 presents the research team’s conclusions 
and recommendations regarding the English Language Acquisition Program. 
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Chapter 2. ELAP Implementation 



Highlights: 

• Approximately 70 percent of the districts in California have at least one EL in 
grades 4 through 8, and are therefore eligible for ELAP funds. Over the first four 
years, the percentage of ELs in districts receiving funds increased from 91 to 98 
percent. 

• Across the first four years of implementation, uses of ELAP funds have varied 
widely, from after- school and Saturday programs to staff development to 
newcomer classes. According to the Year 4 ELAP survey, the most commonly 
reported use of funds is for English Language Development (ELD) instruction. 

• When asked to identify the strengths of ELAP, most districts highlighted the 
focus that ELAP funds placed on English learners. 



This chapter focuses on several aspects of ELAP’ s implementation. First we consider the 
logistics of the program: how eligibility for funds is determined, how districts apply for 
those funds, and when the funds have been made available to districts. Following this 
introduction, we discuss funding patterns, looking at the numbers of districts that were 
eligible and that received funds. Overwhelmingly, the majority of ELs in California 
attend schools in ELAP recipient districts. We then explore the demographic 
characteristics of the districts in the different categories in order to analyze patterns in the 
distribution of ELAP funds. This reveals that ELAP recipient districts tend to have larger 
EL populations and higher poverty levels than non-ELAP recipient districts. Finally, 
based on survey responses, site visits, and phone interviews, we examine how ELAP 
recipient districts allocate and use the funds, and we highlight ELAP strengths as 
articulated by districts. 

The chapter draws heavily on data gathered in the Year 4 survey, administered by the 
American Institutes for Research (AIR)AVestEd and the California Department of 
Education (CDE) to districts that had received ELAP funds. 518 of 546 possible districts 
returned the Year 4 survey, generating a response rate of 95 percent. We also look at 
data gathered in a survey conducted by AIR/WestEd in Year 2, administered to a small 
sample of districts not selected with regard to ELAP participation. Data from phone 
surveys and site visits are included as well. 

Awareness of ELAP at the District and School Level 

Awareness of ELAP at the district and school levels has grown considerably over the 
course of this study. During the first year of the study, a number of districts expressed 
confusion over the availability of ELAP funds and how to allocate them. Quite a few 
individuals, particularly at the school level, were unaware of these funds. By the second 
year of the study, district survey respondents indicated a high level of familiarity with the 
availability of ELAP funds, and an overwhelming majority of districts (92 percent) 
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indicated that they had applied for ELAP funding. Moreover, among respondent districts 
that did not apply, only one was not aware that ELAP funds were available. Continuing 
this trend, all administrators interviewed in the third year of the study were aware of 
ELAP resources. 

Eligibility for ELAP Funds 

The ELAP legislation was designed to give participating districts $100 per English 
learner (EL) in grades 4 through 8. To be eligible for the annual funding, a district only 
needs to meet two requirements: each district must have at least one EL in grades 4 
through 8, and must “conduct academic assessments of English language learners to 
ensure appropriate placement of those pupils.” For the first four years of the program 
(1999-2000 through 2002-03), every district that applied on time (“not later than 60 days 
after the notification from the Superintendent of Public Instruction” as specified by the 
bill) and met these two requirements was given the full funding amount. In addition, 
some districts filing after the deadline also received funding. To determine which districts 
missing the deadline would be given funding, the CDE calculated the number of grades 4 
through 8 ELs as a percentage of total enrollment (in all grades) for each school in the 
applying district. Each individual school was then ranked, and funding was distributed 
first to the schools with the highest percentages of grades 4 through 8 EL students. 
Therefore, in some of the districts applying late, not all of the schools with EL students 
received ELAP funds. 

The number of districts receiving funds grew at an increasing rate over the program’s 
first four years, from 403 in 1999-2000 to 516 in 2002-03 (see Exhibit 2-1). In 2003-04, 
funding was insufficient to cover all the districts applying by the deadline. Because the 
legislation specified $100 per student, the CDE could not simply distribute a reduced 
amount to the entire pool of districts that applied on time. Therefore, this year, for the 
first time, the eligibility formula was applied to districts filing on time. The cut-off point 
for funding, determined by the amount of money available and the number of districts 
applying, constituted 5.46 percent. In other words, in the districts that applied in 2003-04, 
all schools whose grades 4 through 8 EL population was 5.46 percent or more of the 
school’s total student population qualified to receive $100 per EL student. As of this 
writing, letters had been sent to districts notifying them of the amount of funds they 
would receive, but the funds had not yet been disbursed. 

Exhibit 2-1 shows how much funding was provided throughout the state in the program’s 
first four years, and shows the number of districts that received funds. As shown, $53.2 
mi llion was allocated for ELAP in all years of the program except in 1999-2000 and 
2000-01. In 1999-2000, $50 million was initially allocated, followed by an additional 
$1.8 million to fund a small number of districts that had previously been deemed 
ineligible. In 2000-01, an extra $16.8 million was allocated. This amount was earmarked 
to cover a provision of AB 1116 that specified a one-time payment of $100 per student 
for every EL in kindergarten through grade 12 who was “reclassified to English-fluent 
status.” However, the bill required that the California English language development test 
(CELDT) be used to determine English fluency, and the test was not available when 
districts applied at the end of the 1999-2000 school year. 
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Exhibit 2-1. Amount of ELAP Funds Received by Districts 



School Year 


Total ELAP Funds 
Allocated 


Total Amount 
Distributed Statewide 


Number of Districts 
Receiving Funds 


1999-2000 


$51,766,000 


$51,766,000 


403 


2000-2001 


$70,000,000 


$53,796,000 


415 


2001-2002 


$53,200,000 


$53,200,000 


460 


2002-2003 


$53,200,000 


$53,200,000 


516 


2003-2004 


$53,200,000 


N/A 


N/A 



Charter schools were also eligible for ELAP funds. Under the ELAP legislation, charter 
schools could apply directly for funding regardless of whether the school was under a 
district’s administration. Exhibit 2-2 shows how many of California’s charter schools 
applied directly for ELAP and how many applied through their district in the program’s 
first three years. As the exhibit shows, the number of charter schools applying for funds 
directly was relatively low. 



Exhibit 2-2. Number of Charter Schools Applying for ELAP 



School Year 


Total Number of 
Charter Schools 


Charter Schools 
Applying Directly 
for ELAP 


Charter Schools 
Applying for 
ELAP through 
District 


Charter Schools 
Not Receiving 
ELAP 


1999-2000 


154 


5 


99 


50 


2000-2001 


307 


16 


180 


111 


2001-2002 


241 


16 


161 


64 



The Application Process and Timeline 

The CDE sends grant application letters to districts in spring or summer of each year to 
announce the availability of ELAP funds and to solicit applications. The exception is the 
first year’s announcement, which was not sent until December 1999 (ELAP was not 
passed into law until July of 1999; it took until December to establish the initial grant 
application process). The application paperwork is straightforward, and is only four pages 
long. However, it does require that districts applying for the first time submit plans for 
collecting data on measures of EL success and improvement. Once the applications are 
received and complete, and funds are approved, the CDE sends a grant award letter to 
districts notifying them that they were funded and the amount of the grant award. The 
grant award letter gives them the authority to begin spending the funds. Each year some 
districts submitted incomplete applications, typically lacking the required approval from 
their governing board, which delayed their receipt of funds. Their applications were 
processed in two additional funding groups. 

The delay in establishing the initial application process had implications for the 
availability of funds. For the first year of the program, districts received grant award 
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letters in March and did not receive notification of apportionment of funding until May. 1 
In other words, notification regarding Year 1 funds, intended for use in the 1999-2000 
school year, did not arrive until May 2000, two-thirds of the way through the school year. 
Exhibit 2-3 is a timeline showing when districts received the grant award letters and 
notification of funding apportionment for each of the first four years of ELAP funds. The 
statistical analysis in this report is based on the dates on which the first apportionment 
letters were sent. As the exhibit shows, the Year 3 and Year 4 notifications were sent in 
the late fall of the appropriate year, allowing districts time to integrate the funds into the 
budget and curriculum for that school year. The exhibit also shows that the SBX1 18 bill 
was signed into law in March 2003. This bill gave school districts the flexibility to use 
ELAP funds for purposes other than instruction, an added complexity discussed later in 
this chapter. The delayed availability of funds during the first year of the program also 
has implications for the analysis of student outcomes, which will be discussed in depth in 
Chapter 4. 



1 It is difficult to determine whether districts began using ELAP funds upon receiving the grant award 
letter, or when notification of apportionment arrived. From a research perspective, however, using the date 
of the apportionment letter (rather than assuming activity dating from the grant award) to examine the 
effect on achievement relies on fewer assumptions, making it a more conservative approach. 
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Exhibit 2-3. ELAP Funding Notification Timeline 



ELAP enacted 


Year 1 


Year 2 


Year 3 


Year 4 




(AB 1116) 


Grant Award 


Grant Award 


Grant Award 


Grant Award 


SBX1 18 


July 1999 


March 2000 


November 2000 


September 2001 


October 2002 


March 2003 




1 st Apportionment 1 st Apportionment 1 st Apportionment 1 st Apportionment 

May 2000 March 2001 October 2001 November 2002 
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Funding Patterns Over Time 

As discussed in the previous section, there are two eligibility requirements for a district to 
receive ELAP funds, as specified by the text of AB 1 1 16: districts must have at least one 
English learner enrolled in grades 4 through 8, and they must “conduct academic 
assessments of English language learners to ensure appropriate placement of those 
pupils.” This section takes a closer look at the districts that were eligible for ELAP funds 
on the basis of the number of ELs in grades 4 through 8. 2 It identifies the percentage of 
eligible districts that received ELAP funds. Exhibit 2-4 presents the percentage of 
districts in California that were eligible for ELAP funds from 1999-2000 through 2002- 
03. 3 



Exhibit 2-4: Percentage of Districts That Were Eligible For ELAP Funds (N=1 ,091 ) 

100% 

90 % 

80 % 

70 % 
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40 % 

30 % 

20 % 

10 % 
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Exhibit reads: 73.3 percent of districts were eligible for ELAP funds in 1 999-2000. 









73.3% 




71 .0% 




71 .5% 




72.0% 







































































1999-2000 2000-2001 2001-2002 2002-2003 



2 All districts had the option to conduct an academic assessment of their ELs and fulfill the second 
requirement; therefore, fulfilling this requirement was a post-funding obligation rather than an eligibility 
requirement. As a result, the only requirement that will be considered in this report to determine eligibility 
for ELAP is whether districts have at least one EL in grades 4 through 8. 

1 The 15 charter schools that applied and received funds in 2001-02 separately from their district of 
residence are included in the analysis throughout Chapter 2 as part of their district of residence. 
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In the first four years of the program, a relatively consistent 71 to 73 percent of the 
districts in California had at least one EL in grades 4 through 8, thereby being eligible for 
ELAP funds. Below, Exhibit 2-5 shows the percentage of English learners in districts 
receiving ELAP funds in each year. Note that the percentage of grade 4 through grade 8 
ELs in districts receiving ELAP funds increased from 1999-2000 to 2002-03 by 7 
percentage points, from 91 to 98 percent. 

Exhibit 2-5: Percentage of English learners by Receipt of ELAP funds (N=1,091) 




□ Percentage of grades 4-8 ELs in districts that did not apply for EL funds 

□ Percentage of grades 4-8 ELs in districts that received ELAP funds 



Exhibit reads: In 1999-2000, 90.6 percent of grades 4 through 8 ELs were in districts that 
received ELAP funds and 9.4 percent were in districts that did not. 
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While Exhibit 2-5 shows that the overwhelming majority of EL students in grades 4 
through 8 were enrolled in districts that received ELAP funds, not all eligible districts 
ended up receiving ELAP funds. According to the CDE, all districts that applied for 
ELAP annual funding (vs. the one-time redesignation funding) in the first four years of 
the program received at least a portion of the funds that they were entitled to. 4 Therefore, 
it follows that eligible districts that did not receive ELAP funds did not apply. Exhibit 2-6 
presents the percentage of districts that were eligible for ELAP funds but did not apply. 
The graph also shows the percentage of districts that received ELAP funds and the 
percentage that were not eligible. It shows that, in the 1999-2000 school year, only 40 
percent of the districts in California received ELAP funds. However, as shown in Exhibit 
2-5, those districts enrolled 91 percent of the state’s ELs in grades 4 through 8. In the 
2002-03 school year, 47 percent of the districts of California participated in the program, 
representing 98 percent of the total population of English learners in grades 4 through 8. 



Exhibit 2-6: Percentage of Districts by Eligibility and Receipt of ELAP Funds (N=1,091) 
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26.7% 




29.0% 




28.5% 




28.0% 
























33.2% 






33.0% 




29.3% 




24.7% 
























47.3% 


42.2% 




40.1% 










38.0% 

































1999-2000 



2000-2001 



2001-2002 



2002-2003 



□ Districts not eligible 
for ELAP funds 



□ Eligible districts that 
did not apply 



□ Eligible districts 
receiving ELAP 
funds 



Exhibit reads: In 1 999-2000, 26.7 percent of districts were not eligible for ELAP funds, 33.2 
percent were eligible but did not apply, and 40.1 percent received ELAP funds. 



4 The districts that received partial funding had schools that were not funded due to the priority formula 
(see chapter 1); however, all districts that applied had at least one school that was funded. The only year in 
which some districts applied but did not receive any ELAP funds is 2003-2004, which is not included in 
this analysis due to the unavailability of data the time of this report. 
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In the Year 2 survey, 8 percent of the respondents indicated that they did not apply for 
ELAP funds that year. One reason given by eligible districts for not applying was that 
they had been unaware of the existence of ELAP funds. No district indicated that they did 
not apply because the application process confused them, which was the other option on 
the survey. 

In the first year of the program, about 40 percent of all districts in California received 
ELAP funds. This is more than half of the districts that were eligible for ELAP. Over the 
next two years, the percentage of districts that received ELAP funds increased and the 
percentage of districts that did not apply decreased. From 1999-2000 to 2002-03, the 
number of districts that did not apply for ELAP funds had decreased by about one-fourth 
(from 33.1 percent of all districts to 24.7 percent). The following section explores district 
characteristics, such as the number of ELs and total enrollment, which may affect a 
district’s decision to apply for ELAP funds. 

Districts’ Demographic Characteristics 

This section compares the demographic characteristics of districts that received funds in 
2001-02 to districts that were eligible but did not apply for ELAP that year. 5 The analyses 
are based on data available from the state, such as the R-30 Language Census, the 
California Basic Educational Data System (CBEDS), and the 2001-02 Common Core of 
Data. These data sources provide information on the number of ELs, the total enrollment, 
and the number of students receiving free or reduced price lunches in California school 
districts. Data regarding the amount of ELAP funds that were received by districts in 
each year were provided by the California Department of Education. 

460 districts received ELAP funds in 2001-02. The average number of ELs in grades 4 
through 8 for these districts was 1,148 students. 6 There were 320 districts that were 
eligible for, but did not apply for, ELAP funds in 2001-02. The average number of ELs in 
grades 4 through 8 in these districts was 55. As expected, districts with higher numbers of 
ELs are much more likely to apply for ELAP. While the application process is not 
difficult, districts that do not have a large number of ELs in grades 4 through 8 may feel 
that the relatively small amount of money available is not worth the effort. 

An analysis was performed to assess the number of districts that could have received 
$50,000 in a given year but did not apply for ELAP funds. Exhibit 2-7 shows the number 
of districts that did not apply, and the amount of ELAP funding they would have been 
eligible for, by year. Twenty-three of the districts that could have received $50,000 or 
more in ELAP funds in 1999-2000 did not apply. This number had dropped to only one 
district in 2002-03. The initial number of districts eligible for significant funding but not 



5 Although information about ELAP recipients and funds is available for the 2002-03 school year, this 
analysis only includes information through 2001-02 in order to be consistent with the analysis of academic 
achievement impact in Chapter 5. 

6 When applying for 2001-02 ELAP funds, the Language Census counts from 2000-2001 are used. Lor 
districts that received ELAP funds but did not have their ELs recorded in the Language Census, the number 
of ELs was estimated by dividing the amount of ELAP funds received by 100 (the dollar amount that was 
received per EL student). 
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applying for ELAP could be attributed to a lack of knowledge about the program in the 
first year, as the majority of those districts applied for ELAP funds in the following year 
( 2000 - 01 ). 

Exhibit 2-7. Districts Eligible for At Least $50,000 in a Given Year That Chose Not 
to Apply for ELAP Funds 



School Year 


Eligible for 
$50,000 to 
$100,000 
ELAP funds 


Eligible for 
$100,000 to 
$150,000 
ELAP funds 


Eligible for 
more than 
$150,000 ELAP 
funds 


Total 


1999-2000 


14 


3 


6 


23 


2000-2001 


5 


0 


2 


7 


2001-2002 


5 


1 


0 


6 


2002-2003 


1 


0 


0 


1 



We then specifically examined the twelve districts that did participate in ELAP’s initial 
year, but chose not to participate for at least one subsequent year. Nine of these districts 
received less than $50,000 in their initial year of funding; this lack of a “critical mass” of 
funding may explain why they did not reapply. Each of the twelve districts appear to have 
experienced an increase in their EL population in the time following the 1999-2000 
school year, and each subsequently reapplied for ELAP funds after a one or two year 
lapse of participation, with all but two of the districts receiving at least $50,000 in 
funding from 2000-01 onward. 

Based on phone calls to six of these districts, two did not know why their district had not 
applied because the person responsible for district ELAP implementation had changed (in 
one case, the job had been eliminated, with responsibilities absorbed by other staff 
members). One district reported that they did not have the application ready before the 
deadline. Although not reflected in the data provided to us by the state, the final three 
districts believed that they had in fact received ELAP funds in the year in question. 

Student attributes in ELAP and non-ELAP districts also differed somewhat. The average 
percentage of ELs in districts that received ELAP funds in 2001-02 was 30 percent, as 
compared to 14 percent in non-applying districts. 

Relative poverty levels in districts that did and did not apply for ELAP funds were also 
analyzed. For the purposes of this study, poverty is measured by the number of students 
eligible for the National School Lunch Program (NSLP) as a percentage of the total 
number of students in the district. The average poverty level in districts that received 
ELAP funds was 38 percent, as compared to 31 percent in non-ELAP districts. 

Fund Allocation Process at the District Level 

In our Year 4 survey, we asked districts to answer several questions regarding their use of 
ELAP funds. 68.9 percent of districts reported allocating some or all of their ELAP 
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funding to school sites, while 31.1 percent of districts did not allocate any funds directly 

•j 

to schools. In our Year 2 survey, a much smaller 52.3 percent of districts reported 
ELAP allocations directly to schools. For both of these years, it appears that a substantial 
number of districts used ELAP funds at the district level rather than allocating them 
directly to schools. However, school-based allocations seemed to increase over time. 

Because ELAP funding is allocated on a per pupil basis, districts without substantial 
numbers of qualifying ELs receive smaller ELAP allocations, which may hamper the 
individual school’s ability to implement ELAP-funded interventions. For districts 
choosing not to allocate funding to schools, the two explanations given were that the 
district only had a single school (funds could not be allocated separately to the school), 
and that due to the small amount of ELAP funds received by the district, it made more 
sense for the district to coordinate the purchase of materials or to run a central 
intervention for ELs. As one respondent explained, “ELAP allocations to our elementary 
schools, based on enrollment of EL students in fourth and fifth grades, would rarely result 
in as much as $3,000. Taken together, elementary allocations district-wide usually total 
$40,000. Therefore [in some situations] we take the opportunity to centralize programs 
and services, which ultimately reaches more students than the fragmented resources to 
individual sites.” 

Some districts commented that they preferred to use larger and more predictable sources 
of funding to run their core English learner programs, and use ELAP funding to 
supplement these programs or coordinate supplemental interventions to target ELs in 
grades 4 through 8 — the federally funded Title I and Title III programs and the state- 
funded English Language Intensive Literacy Program (ELILP) were cited as examples. 

In fact, 37 districts (7 percent) identified the use of ELAP in conjunction with other, 
larger funding streams as the primary difficulty in isolating the effect of ELAP funds on 
students’ improvement on the CELDT and on academic benchmarks, or in evaluating the 
effectiveness of ELAP. 

For districts that do allocate ELAP funding to schools — approximately two-thirds of the 
districts in the Year 4 survey — 173 districts (51.0 percent) spent some percentage of the 
funds centrally while 166 districts (49.0 percent) allocated all ELAP funding directly to 
schools. Looking at the distribution of ELAP funds across these districts shows that 
school sites receive the majority of ELAP funding (86.8 percent). 

When comparing the allocation patterns of districts that do and do not allocate ELAP 
funding to schools, the average percentage of ELAP expended on most categories of 
programs/expenditures is comparable (Exhibit 2-8). However, districts allocating 
directly to schools show statistically different allocations in the areas of core academic 
instruction, extended time programs, and language testing and assessment. These 
variations in allocation may be because core academic and extended-time programs are 
likely to be site-based, while language assessment and testing is more likely to be 
conducted centrally. 



7 The Year 2 survey had a 60 percent response rate, as well as a smaller sample size. Further details are 
provided in the Year 2 Report, available online at www.wested.org/cs/we/view/rs/661 . 
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Exhibit 2-8. Percentage of ELAP Funds Allocated to Different Categories for Districts that 
Did or Did Not Allocate Funds to Schools (N=518, 18 responses missing) 




Core Academic ELD Instructional Extended Time Newcomer Staff Language Other Programs 

Instructional Programs Programs Services Development Assessment and 

Programs Testing 



□ Allocate to Schools 



□ Do Not Allocate to Schools 



Exhibit reads: Districts that allocate their ELAP funds to their schools spent an average of 1 1 .7 
percent of those funds on core academic instructional programs; districts that do not an allocate 
to their schools spent an average of 7.4 percent of their ELAP funds on core academic 
instructional programs. 



The majority of districts (68.9 percent, or 236 districts) allocating ELAP funding to 
school sites do so on the basis of a fixed amount per EL. 9.0 percent of districts distribute 
a fixed amount per school or type of school. 5.6 percent of districts reported allocating 
ELAP on the basis of school budgets. The remainder (18.2 percent, or 65 districts) 
assigned ELAP funding on some other basis. 



How districts and schools use ELAP funds 

In the first year of our evaluation, we conducted phone interviews and site visits to gather 
information about Proposition 227 and ELAP. These occurred during the first year that 
districts actually received ELAP funds (2000-01), so we were able to explore how 
districts and schools had begun to strategize about using the funds, and how they were 
initially choosing to allocate the money. We found that some schools and districts 
employed needs assessments to gather input about potential uses of the funds. These 
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included internal meetings, consultations with school site councils, surveys, and open 
meetings to which parents were invited and encouraged to participate and provide input. 
Actual uses of the funds varied widely, as would be expected given the latitude of the 
legislation. Uses identified during the 2000-01 school year included the following: 

• After-school and Saturday programs 
. Staff development 

• Planning time (paying stipends to teachers) 

• Support/resource materials 

• Transitional reading programs, listening centers, computer programs 

• Payments for substitute teachers so regular teachers could provide one-on-one 
help to students in need 

• Sending teachers to conferences 

• Intersession instructional programs 

• Newcomer classes 

• Summer reading camps for ELs 

• Language assessments and redesignation testing 

• Instructional assistants 



We also asked about the most common uses of ELAP funds in the Year 4 survey. The 
survey was sent during the fall of the 2003-04 school year, after districts had received up 
to 4 years of ELAP funds. In the survey, we asked which programs and services were 
being supported by ELAP. As Exhibit 2-9 shows, the most common reported use of 
ELAP funds in the Year 4 survey was for ELD instruction (78.6 percent). Extended-time 
programs, language testing and assessment, and staff development were also frequent 
areas for ELAP spending. 
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Exhibit 2-9. Percentage of Districts Using ELAP Funds for Various Purposes (Year 4 
Survey, N=518, 8 responses missing) 




ELD instructional Extended time Language testing Staff Core academic Newcomer Other 

program program(s) and assessment development instructional services 

program 



Exhibit reads: 36.5 percent of districts reported using ELAP funds on core academic programs in 
the Year 4 survey. 



We asked a similar question in the Year 2 survey. While the two questions are 
sufficiently different that they cannot be compared, some differences can be observed. In 
Year 4, the most commonly reported use was “ELD instructional program.” In the Year 2 
survey, “ELD instructional program” only ranked fifth, behind four other uses. The most 
common was “resources or materials,” followed by “extended time program(s)” and 
“staff development.” 

In the Year 4 survey, we asked respondents to tell us the percentage of total ELAP funds 
used for various purposes (Exhibit 2-10). Districts using ELAP funds for ELD instruction 
(77.6 percent) spent an average of 46.2 percent of their ELAP budget on those activities. 
Extended time programs received the second-highest percentage at 39.5 percent. No other 
program or service received more than an average of 30 percent of ELAP spending. 
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Exhibit 2-10. Average Percentage of District’s Total ELAP Funds Used in Each Category 
That Received ELAP Support (Year 4 Survey, N=518, 8 responses missing) 




ELD instructional Extended time Other Core academic Language testing Staff Newcomer 

program program(s) instructional and assessment development services 

program 



Exhibit reads: Districts that used ELAP funds for core academic instructional programs allocated 
an average of 27.6 percent of their ELAP funds to that category according to the Year 4 survey. 



We also asked Year 4 survey respondents to report the percentage of ELAP funds spent 
on alternative types of resources. As shown in Exhibit 2-11, about a third of the surveyed 
districts’ ELAP funds were spent on certificated personnel (34.5 percent), and another 
third on textbooks, materials and supplies (31.7 percent). Information gathered from our 
site visits and phone interviews suggests that the most common use of the ELAP funds 
spent on certified personnel was to pay existing personnel to staff extended-time 
programs (after-school, Saturday, and summer school programs). In the Year 4 survey, 
one district commented that “all of our programs are directed and taught by existing 
classroom teachers. ELAP provides the financial incentive to stretch them a little farther 
and a little longer (after school and summer school).” 
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Exhibit 2-11. Average ELAP Fund Distribution Across Categories of Expenditure (Year 4 
Survey, N=518, 4 responses missing) 




Certificated personnel Textbooks, materials, Non-certificated Other Technology or 

supplies personnel equipment 



Exhibit reads: Districts reported using an average of 34.5 percent of their ELAP funds on 
certificated personnel on the Year 4 survey. 



A number of districts provided additional specifics on their ELAP-funded efforts, 
including CLAD, BCLAD, SB395, SDAIE, and early literacy training. One district gave 
the following description: “we have an ongoing professional development plan for 
teachers of EL students that includes both local and regional presenters, curriculum-based 
training, site visits to other programs, and ongoing support by the bilingual deputies in 
the Language Development Office.” Highlighting the success of their staff development 
efforts, another district stated that “both administrative and instructional staff. ..are much 
more able to identify areas of need, as well as strength, and are better able to identify the 
type of support needed.” 

Districts were also asked to identify the strengths of ELAP. Most districts highlighted 
the focus that ELAP funds placed on English learners. The expansion of resources 
targeted to English learners was cited as extending beyond the increased availability of 
EL interventions and support services. One district commented, “following the progress 
of individual students has promoted parent, student, and teacher awareness of the need to 
enhance academic language applications.” In this same vein, another district remarked, 
“Each time new dollars are earmarked for English learners, renewed emphasis is placed 
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on their progress, both in the minds of teachers and administrators.” Similarly, rewriting 
of district EL plans, an increased sharing of information and test data between staff and 
parents, and a greater focus on ongoing assessments of both core curriculum and ELD 
were also singled out as ELAP-related changes in participating districts. 

Professional Development Institutes 

It should be noted that the ELAP legislation also provided for the establishment of 
professional development institutes. These were designed to improve teachers’ ability to 
help English learners in grades 4 through 8 meet state content standards in English 
language arts, mathematics, science, and history/social science. The institutes were 
established jointly by University of California Office of the President (UCOP), the 
Chancellor of the California Community Colleges, the Independent Colleges, and the 
Superintendent of Public Instruction, and were administered by UCOP. Service was 
provided by consortia of college-based organizations (primarily the California Subject 
Matter Projects) and collaborating school districts. Each professional development 
program focused on English language development and academic English development 
within the context of one of the four content areas (English language arts, mathematics, 
science, and history/social science). All programs consisted of an intensive institute 
lasting at least 40 hours (usually offered during the summer) and an additional 80 hours 
of instruction and school-based activities over the course of a school year. Priority for 
participation was given to (1) teachers who did not hold CLAD/BCLAD certificates or 
their equivalents, (2) staff teams from low-performing schools, (3) schools with average 
reading scores below the 40 th percentile on the English language arts portion of the 
Stanford 9 test, (4) schools in which enrollments of English learners exceeded 25 percent, 
and (5) schools funded by ELAP. 

During 2000-01, the initial year of operation, funds were authorized to serve 5,000 
educators statewide. A year later, additional legislation (AB 8221) expanded the scope of 
the institutes to include grades kindergarten through 12, consolidated them with the 
California Professional Development Institutes (which also included professional 
development programs in K-6 reading instruction, high school English language arts, and 
mathematics), and provided for the training of an additional 10,000 participants. 

However, funding for the institutes was eliminated from the 2002-03 budget, effectively 
ending the program that AB 1116 had initiated. According to UCOP estimates, a total of 
about 8,000 teachers attended ELD institutes between 2000 and 2003. 

Conclusion 

This chapter has detailed the implementation of ELAP within California’s school 
districts. We have shown that awareness of ELAP has grown over the course of the study, 
and that the number of districts receiving funds grew at an increasing rate over the 
program’s first four years. And while this was less than half of California districts in 
2002-03 (47 percent), almost all of the state’s EL students (98 percent) were in those 
districts. Survey data indicate that the most common use of ELAP funding was for ELD 
instruction, followed by an extended time program or programs. The timeline for ELAP 
fund disbursement reveals that the first and second years of funding arrived late in the 
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school year, complicating implementation. Many districts considered the availability of 
funds to expand supplemental interventions and programs for EL students in grades 4 
through 8 ELs to be a significant strength of the program. The remaining chapters will 
consider challenges to the implementation and evaluation of ELAP by districts, will 
examine the impact of ELAP through analysis of statewide data, and will offer 
conclusions and suggestions for improvements to the program. 
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Chapter 3. Implementation Challenges and District 
Evaluations of ELAP 

Highlights: 

• The restriction of ELAP funds to grades 4 through 8 was reported as the biggest 
constraint in the Year 4 survey. However, some districts saw this restriction as a 
strength. The second most common constraint cited was uncertainty about 
whether funds would be available. 

• Despite a requirement to do so, only 5 percent of the Year 4 survey respondents 
(26 respondents) indicated that they had conducted a formal evaluation of ELAP. 
Only 7 of the 26 evaluations explicitly discussed the role of ELAP in the district. 

• Five districts responding to the Year 4 survey attached evaluations that reflected a 
clear effort to evaluate the impact of ELAP on student achievement. Their 
analysis plans used measures such as statewide tests to follow the academic 
achievement of ELs or qualitative indicators of student progress. 

• Many districts noted on the Year 4 survey that they were not able to judge the 
impact of ELAP because they do not specifically evaluate ELAP. However, more 
than half the districts reported the belief that ELAP funds had had a moderate to 
large impact on areas of language development, academic achievement on state 
content standards, and school redesignation rates. 

• Attempting to assess the impact of a state program like ELAP based on the 
independent efforts of individual school districts appears unrealistic. 

Chapter 3 discusses the challenges faced by districts implementing ELAP programs, then 
examines the ELAP district evaluations mandated by AB 1 1 16. We begin the chapter by 
considering the context for ELAP implementation, and then review the challenges 
reported by the districts that have received ELAP funds. The most common challenges 
cited were the restriction to grades 4 through 8 and uncertainty regarding the availability 
of funds. Following this contextual introduction, district ELAP evaluations are reviewed 
as required by AB 1116. Twenty-six districts submitted such evaluations, and only seven 
focused on an approach that established a link to ELAP. Based on these submissions, 
attempting to isolate the potential impact of a funding stream like ELAP is beyond the 
evaluative capacity of most school districts. 

Issues and Challenges in Implementing ELAP 

Districts and schools have faced many obstacles in implementing ELAP. One obvious 
initial difficulty, as discussed earlier in this report, was that districts had to wait until late 
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in the school year for funds to be apportioned in both the first and second years. 1 And 
now, in the 2003-04 school year, Year 5 funds will be arriving months later in the school 
year than Year 3 and Year 4 funds did (October and November, respectively). 

Another challenge has been created by two recent bills that affect how ELAP funds may 
be used. Senate Bill XI 18 went into effect on March 18, 2003. This legislation, which 
was a response to California’s education budget cuts, allowed districts to redirect money 
from restricted general fund programs, including ELAP, to other purposes. Under the bill, 
ELAP funds that had not yet been spent were no longer restricted to EL-related activities. 
The legislation added a section to the California education code stating that an LEA could 
use “up to 50 percent of the balances, as of July 1, 2002, of restricted accounts in its 
General Fund. . . in order to provide local budgeting flexibility as a result of midyear 
budget reductions for the 2002-03 fiscal year. . .” (California Education Code Section 
33128.2). While the total amount was capped at 50 percent, there was no such restriction 
on the amount pulled from individual accounts, including ELAP. In other words, a 
district could use 100 percent of any ELAP funds carried over from the Year 1, Year 2, 
and Year 3 fund disbursements for purposes unrelated to EL education. The legislation 
expired after one year, but AB 1754 was passed in 2003-04 to allow similar budget 
flexibility. While this flexibility may assist districts feeling the effects of budget cuts, the 
ability to use ELAP funds for other needs potentially reduces its impact. 

In site visits and phone interviews in the first three years of this study, district and school 
personnel generally expressed appreciation for ELAP funds. At the same time, logistics 
were often mentioned as a major challenge in implemention. Finding available teachers, 
space, and transportation affected efforts by many schools to create after-school, 

Saturday school, and intersession programs for ELs. In the first and second years of the 
study, districts also mentioned that the lack of appropriate materials designed for ELs 
made implementation difficult. 

The Year 4 survey asked districts to what extent various factors were constraints on their 
ability to use ELAP funds (Exhibit 3-1). Restriction of funds to grades 4 through 8 was 
viewed as the biggest constraint, with 65.5 percent of districts referring to it as moderate 
or large. On our Year 2 survey, participating districts responded similarly to a 
comparable question. The Year 4 survey also asked respondents to write in any other 
issues they had encountered in implementing ELAP. Several districts took this 
opportunity to explain that the majority of their EL students were in kindergarten through 
grade 3 and therefore not eligible to receive ELAP funds. Other districts pointed out the 
restriction of ELAP resources to certain grades (especially for elementary districts) 
complicated the design of interventions, and suggested that funds could be more 
effectively used at the district’s discretion, given that their “preference would be to serve 
students based on most in need, not on grade level.” 



1 As previously noted, it is difficult to determine whether districts began using ELAP funds upon receiving 
the grant award letter, or when notification of apportionment arrived. From a research perspective, 
however, using the date of the apportionment letter (rather than assuming activity dating from the grant 
award) to examine the effect on achievement relies on fewer assumptions, making it a more conservative 
approach. 
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However, this was not the perception across all districts. Seventeen districts included 
detailed comments explaining why the restriction to grades 4 through 8 was, in fact, a 
strength of ELAP in their schools. These districts pointed to their increased ability to 
“target the grade span that is most at risk of not meeting academic standards” and to 
direct services at ELs “plateauing at an intermediate level of English” by supporting 
those ELs in the larger class sizes encountered in the later grades, by continuing to focus 
on the literacy development begun prior to grader 4, and by allowing closer monitoring of 
individual student achievement to ensure mastery of grade level standards. One district 
said that the success of their ELAP Saturday ELD classes prompted them to use their 
federal funds to start an additional first- through third-grade Saturday class. 



Exhibit 3-1. Constraints to Utilizing ELAP Funds (Year 4 Survey, N=518, 0 missing 
responses) 




Restriction of Uncertainty of Delayed Lack of Lack of Lack of Lack of Other 

funds to available funds receipt of appropriate EL guidance on qualified classroom 
grades 4-8 funds instructional how funds can teachers space 

materials be used 



Exhibit reads: 65.5 percent of districts reported that restriction of ELAP funds to grades 4 
through 8 was a constraint, to a “moderate” or “large” extent, to using ELAP funds on the Year 4 
survey. 



The second most common constraint cited in the Year 4 survey was uncertainty about 
whether funds would be available, reported by 57.1 percent as moderate or large (Exhibit 
3-1). Several districts commented that this uncertainty was particularly problematic when 
trying to hire staff. Related to uncertainty over funding availability, 30.2 percent of 
districts cited delayed receipt of funds as a constraint to a moderate or large extent. 
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Seventeen districts specifically mentioned that these delays hindered their ability to plan 
ahead or implement programs. One respondent commented that, “The funding is 
temporary, which limits capacity building. If schools knew for sure that funds would be 
available at a minimum for a specific number of years, they could create more viable 
short-term support systems. Annual notification of funding does not encourage 
thoughtful planning and analysis.” 

Flexibility in the use of ELAP funds was another aspect of the program that elicited 
varied responses from districts. The flexibility, which allowed funds to be adapted to the 
specific needs of the individual site, was singled out as a benefit — 19 districts specifically 
mentioned the program’s flexibility as a strength. However, district comments also 
emphasized the desire for more concrete guidance, with 19.2 percent of district 
respondents listing a lack of guidance on how funds can be used as a moderate or large 
constraint (Exhibit 3-1). Several districts requested further information about successful 
ELAP programs. One district suggested that “having a more complete and specific 
program guideline package would enable the district to evaluate the program specifically 
aligned to the goals of ELAP.” 

Another issue related to flexibility cited by a few districts is that “indirect costs can only 
be charged for centralized services and activities that are requested by the schools 
receiving the funds.” Several districts reported that the lack of accompanying 
administrative funding made adequate program oversight challenging at the district level. 
One reported, “As the funding was to go directly to schools and not funneled through a 
central office, schools did not provide standardized or coordinated services through 
ELAP funding. Each school used funds to provide services they thought appropriate, 
including social-emotional student and parent services, academic materials, instruction, 
and staff development training. However, the ELAP programs varied widely from school 
to school, in both quantity and quality of services.” 

Finally, while not specifically a constraint to ELAP implementation, based on district 
write-in comments, the introduction of the CELDT may have also challenged districts’ 
intended use of ELAP. Districts repeatedly cited the value of the CELDT as an 
assessment to determine the placement of students and the services they receive, but 
voiced concerns about the cost of the test. One district reported, “The present level of 
remuneration from the state has led many districts to use ELAP money to help defray the 
cost of administration for this test. . . Since the CELDT is a mandated cost, it has led to a 
net reduction in services available to English learners.” Districts were reimbursed at a 
rate of $1.50 per test for CELDT testing between May and October of 2001, with the 
apportionment rate increasing to $5 beginning in November of 2001. While the actual 
cost of the test to the district depends on factors such as who proctors the exam (district 
teachers or outside staff) and where the test is scored (onsite or by the test publisher), one 
cost estimate by a source from the Standards and Assessment Division of the CDE is 
$15-$20 per test. In the Year 4 survey, six districts described the need to use ELAP funds 
to help cover the costs of English language assessment as something that reduced their 
ability to fund direct services for EL students. 



2 See http://www.cde.ca.gov/cilbranch/elap/elapfaq.html 
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District Evaluations of ELAP 

As described in Chapter 1, ELAP legislation required each receiving district to evaluate 
its effectiveness in improving EL instruction and student outcomes. Districts reported 
using a wide array of tools in attempting to measure the effectiveness of ELAP, while 
also noting considerable difficulty in attempting to isolate the program’s impact. The 
assessment most cited by districts was the California English Language Development 
Test (CELDT), with 78.9 percent of the districts so responding (Exhibit 3-2). The 
Stanford Achievement Test, 9th Edition (SAT-9), the California Achievement Test, 6 th 
Edition (CAT-6), and the California Standards Tests (CST) were also cited by more than 
two-thirds of the districts. District writing proficiency tests, other district- wide 
assessments, and non-cognitive indicators such as attendance and retention rates were 
also cited as bases for ELAP evaluation. In addition, 44 percent of district respondents 
indicated that their district used the observation of teacher practice to assess the 
effectiveness of ELAP. Teacher surveys were also used by a limited number of districts 
(19.6 percent). 



Exhibit 3-2: District Reported Measures Used to Assess the Effectiveness of ELAP (N=518, 
21 missing responses) 



Percent Yes 


California English Language Development Test (CELDT) 


78.9% 


California Standards Test (e.g., English-Language Arts, Mathematics, History/Social 
Studies, Science) 


73.6% 


SAT -9/CAT -6 


72.5% 


District writing proficiency test 


55.9% 


Other district-wide assessment 


52.4% 


Observation of teacher practice 


44.4% 


Non-cognitive indicators (e.g. attendance, retention) 


23.7% 


Teacher surveys 


19.6% 


Other* 


24.3% 


*Note that only 1 81 out of the 497 districts responded to this item, 24.3% of which responded “yes.” 


Exhibit reads: 78.9 percent of districts reported using the CELDT test to assess ELAP’s 
effectiveness. 
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District ELAP Evaluation Methods 

Although 96 percent of districts indicated that they used at least one of the data sources 
listed above to assess the effectiveness of ELAP, when describing methods used for their 
data analysis, only seven districts outlined an approach that focused on establishing a link 
specific to ELAP. For example, one district described a comprehensive evaluation of 
ELAP that included a review of CST scores, a teacher survey, classroom observations, a 
student survey, and redesignation rates of ELAP students. However, the district 
acknowledged that they did not compare these gains with students who did not participate 
in ELAP activities, and did not attempt to draw any final conclusions about the 
effectiveness of ELAP. Other districts indicated that they attempted to analyze at least 
one of the performance indicators listed above, but did not indicate how this analysis 
related to assessing the effectiveness of ELAP. 

Instead, districts more often indicated that these data were used to target ELAP funds for 
specific program development, to assign students to particular classes or ELAP-funded 
instructional services, and to determine professional development needs of school staff. 
Five districts reported that they regularly sent student achievement reports to schools to 
inform teachers of the progress of ELs in language acquisition and academic 
achievement, and that teachers in turn used these reports to target students for academic 
support funded by ELAP. One district explained: “CELDT scores are used to place 
students in EL programs. Students at the beginning to intermediate level get after-school 
tutoring and summer school paid from ELAP funds. SAT-9/CAT-6 and CST scores as 
well as district writing assessments are used to assign students to after-school 
intervention tutoring and summer school for students at the early advanced and advanced 
levels.” 

Description of District Evaluations 

Of the full set of district respondents, approximately 5 percent (26 districts) indicated that 
they had conducted a formal evaluation of ELAP. Districts so responding were asked to 
attach their evaluation to the survey. For the most part these attachments were simply 
listings of district-wide EL or total student population test scores. Only 7 of the 26 
evaluations explicitly discussed the role of ELAP in the district. Most likely, this is 
because districts did not know how to isolate the impact of ELAP on ELs. For example, 
one district analyzed the performance over time of all ELs and appropriately noted that 
“it would not be prudent to point to one program in particular as the one reason for the 
ELL student growth.” 

Of these seven evaluations that clearly target the role of ELAP, the evaluations of five 
districts reflect a distinct effort to specifically evaluate the impact of ELAP on EL 
achievement. Their analysis plans used measures such as the CELDT, SAT-9/CAT-6, 
CST, and district proficiency tests to follow the academic achievement of the ELs 
participating in ELAP. Four of these districts looked at all ELs in grades 4 through 8 over 
time, while the fifth district created two cohorts of ELAP recipients (in grades 4 and 6) 
and matched their scores over time. 
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While these analyses attempt to use data to assess the progress of ELAP recipients, as 
none include data from a comparison group, no “ELAP effect” can be inferred. For 
example, while one district noted a gain in CST scores for ELs in grades 4 through 8 over 
time, without a comparison group the potential impact of ELAP in relation to everything 
else occurring in the district during this time period cannot be determined (i.e., all of the 
students in the district may have achieved higher test scores over this time period). 

Another district evaluation was based entirely on qualitative information, in which they 
described their system of monitoring ELAP implementation in schools and presented a 
detailed picture of how the ELAP funds were spent in each school. They also included a 
qualitative description of student progress indicators (such as improved redesignation 
rates or CELDT scores), as seen or expected to be seen in the future. However, they 
included no quantitative indicators. 

One last district did appear to have an effective system for monitoring ELAP in their 
schools as well as plans for quantitative evaluation in future years. The qualitative 
analysis described use of ELAP funds in the district, perceived benefits from ELAP- 
funded programs, and suggestions for improving ELAP in the future. The quantitative 
component of the evaluation examined changes in the redesignation rate of all ELs to 
RFEPs from 1998-99 to 2002-03. There were also plans to examine graduation rates in 
2005, when the eighth-grade students who benefited from ELAP funds in the first year of 
the program will be graduating. In addition, they plan to initiate a system in two years 
that will allow the district to distinguish the progress of ELAP recipients versus non- 
participating ELs on ELD tests and the SAT-9. Furthermore, this district has an ELAP 
Coordinator who creates a binder for each ELAP site that includes the following 
components: a site ELAP preliminary budget, general guidelines for ELAP program 
activities and appropriate expenditures, and a sample “End of Year Program Evaluation.” 

Challenges to assessing the impact of ELAP 

These relatively meager results from the required district evaluative effort of ELAP are 
understandable. Trying to isolate the impact of ELAP from the many program initiatives 
and other outside factors occurring during the period of ELAP implementation is quite 
challenging, and is likely to be daunting for individual school districts. This raises 
important questions in regard to the state delegating evaluative responsibilities for major 
funding initiatives to individual school districts. 

Throughout the Year 4 survey, districts reported a number of challenges in their efforts to 
evaluate ELAP. They noted that the interplay of other programs for ELs, as well as other 
school reform efforts, made it difficult to attribute student outcomes to ELAP. Districts 
also reported that because ELAP funds are often combined with other funding sources to 
fund an English language development program, it is difficult to assess the effectiveness 
of ELAP funds alone. One district stated, “It is impossible to assess how much of an 
impact ELAP has had on the CELDT, SAT-9/CAT-6, Standards Tests and other 
measures of EL progress as so many other programs have been in play that affect the 
results: EIA/LEP, Title VII, Title I, IIUSP/High Priority Schools, etc. The results cannot 
be ascribed to any one program, especially a relatively small program such as ELAP.” 
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Two districts also noted that changes and modifications in the state assessments and 
CELDT test materials made it difficult to assess the program over time. In addition to 
changes in the tests themselves, one of these districts reported that it was difficult to 
collect longitudinal information on EL students because many of them move before they 
can be retested. The second district cited the lack of an evaluation model and the inability 
to use ELAP funds for data entry and interpretation as problems. 

Districts’ attempts to track the progress of all ELs 

Although it is difficult to tie EL performance directly to ELAP, six districts reported in 
the Year 4 survey that they were developing comprehensive methods to assess EL 
outcomes. One district noted, “We are developing an Academic Success Index for our 
English Learners that will include CST scores, grades for English and math, promotion, 
California High School Exit Exam (CAHSEE) results for language arts and math, and 
attendance.” Another district requires that all EL students who are participating in 
intervention programs such as ELAP also participate in a “Pass with a plan.” This 
individualized plan “calls attention to their needs, identifies previous assessments 
[including CELDT, SAT-9/CAT-6, CST, district assessments for language arts and math 
and report card grades], lists interventions received and outlines the services needed.” 
Another useful tool in tracking the progress of ELs described by one district is the Title 
III Annual Measurable Achievement Objectives (AMAOs). This district reported using 
the state targets for gains in the number of students who grew at least one proficiency 
level in a year on the CELDT, as well as the gain in the percentage of students attaining 
English proficiency, as a guide for evaluating their own progress. Other districts 
described a less complex approach to analyzing how their ELs are performing; thirteen 
districts simply reported that overall CELDT scores for their districts were rising. 

One district summarized what seemed to be a theme for many by saying that “data 
analysis [suggests] that ELAP, as part of coordinated services to EL students in grades 4 
through 8, is part of an effective program.” While this general sentiment was commonly 
expressed by a number of survey respondents, specific evidence of this was almost 
universally lacking. 

Assessing the impact of ELAP at the district level 

As another part of the Year 4 survey, respondents were asked to describe to what extent 
they believed that ELAP had resulted in outcomes for their district in three areas: 
improved performance on tests assessing ELs’ English language development, improved 
performance on tests assessing ELs’ grade level achievement on state academic content 
and performance standards, and increase in school rates of EL redesignation to English 
fluency. As shown in Exhibit 3-3 below, for each area of possible impact, a fairly large 
number of respondents (from one-fifth to one-fourth in each case) chose the “no basis for 
judgment” response. Many of these respondents noted that their district does not 
specifically evaluate ELAP. 

The majority of districts, however, did submit judgments regarding the impact of ELAP 
in their district. More than half reported ELAP funds to have had a moderate to large 
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impact on areas of language development, academic achievement on state content 
standards, and school redesignation rates. According to district responses, improved 
English language development on EL assessment tests is the area most directly affected 
by ELAP, with 62.1 percent of districts citing a moderate to large impact, 54.5 percent of 
districts cited a moderate to large impact on improved performance on academic content 
and performance standards, and 50.9 percent of districts cited a moderate to large impact 
on their rates of redesignation. 



Exhibit 3-3: Extent to which ELAP has resulted in district outcomes (N=518, 19 missing 
responses) 



Improved English language 
development on EL 
assessment tests 



Improved EL grade level 
performance on state 
academic content/standards 



Increased school rates of EL 
redesignation to English 
fluency 



Other 









0.6% 




15.8% 


46.3% 


14.6% 


22.7% 








1 .0% 




9.5% 


45.0% 


21.4% 


23.2% 








1 .9% 




12.3% 


38.6% 


22.6% 


24.6% 


3.5% 2.6% 










8.6% 


1 0.3% 






75.0% 













0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



□ Large extent □ Moderate extent □ Small extent □ Not at all □ No basis for judgment 



Exhibit reads: 15.8 percent of districts reported that ELAP has resulted in improved English 
language development on EL assessment tests to a “large extent.” 
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The survey also asked districts to present evidence and examples demonstrating the 
improved outcomes that were reported. For improved performance on tests assessing 
English language development, 65 districts detailed their CELDT scores based on a year- 
to-year comparison as evidence for the impact of ELAP funds. In addition to this 
longitudinal two-year comparison, ten districts specified an increased percentage of 
students scoring at the Early Advanced and Advanced levels of the CELDT. In addition 
to CELDT scores, one district commented that teachers reported that students attending 
ELAP programs demonstrated “more oral fluency and class participation.” 

Similarly, 60 districts cited improved SAT-9 and California Standards Test (especially 
Math and English-Language Arts tests) results as evidence of ELAP’ s impact on 
achievement on state academic content and performance standards. Fifteen districts 
briefly discussed the Adequate Yearly Progress (AYP) and Academic Performance Index 
(API) scores for their EL subgroups, with most districts highlighting their growth in the 
past three years. One district reported, “For our API, our Hispanic/Latino population 
[subgroup]... increased by 102 points, well exceeding our target.” 

Some districts mentioned that their ability to perform longitudinal analyses of student 
performance was particularly constrained by the recent changes in tests used in 
California, especially the change from SAT-9 to CAT-6. This change in particular 
prevented longitudinal achievement comparisons across all years that the district received 
ELAP funding. One district mentioned that they had hired an outside assessment 
organization to conduct analyses of their student data. 

Increased redesignation of ELs to fluent English proficient (RFEP) was also cited by the 
ELAP legislation as a basis for evaluating the program. In response to the survey, 41 
districts cited increased redesignation rates as evidence of ELAP’s impact. As additional 
evidence, some districts included a comparison of their redesignation rates to county and 
state redesignation rates, as well as comparisons between different schools within their 
district. 

Conclusion 

When districts were asked what they considered the biggest constraints to implementing 
ELAP, the three most common items were the uncertainty of availability of funds, the 
absence of accompanying administrative funding or concrete guidance, and the restriction 
of funds to grades 4 through 8. (It should be noted that the absence of concrete guidance 
was due to the program’s flexible design, which other districts cited as one of the 
program’s strengths.) Beyond these implementation challenges, a review of recipient 
districts’ responses to the ELAP survey in regard to their attempts to self-evaluate ELAP 
largely points to the difficulties associated with this requirement. Attempting to isolate 
the impact of a funding stream like ELAP from among several other initiatives and 
funding sources appears to be beyond the evaluative capacity of most individual school 
districts. When local evaluation requirements are still considered useful, perhaps for 
purposes of improving implementation, it may be best to provide districts with examples 
of evaluative models or methods. The next chapter provides an attempt to independently 
assess possible ELAP impact statewide. 
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Chapter 4. Analysis of the Impact of ELAP Funds 



Highlights; 

• Multiple regression analyses were used to estimate the relationship between 
ELAP and the academic performance of ELs, while controlling for demographic 
and socioeconomic differences between ELAP and non-ELAP schools. 

• To estimate a possible ELAP “effect” despite data limitations, three different 
combinations of EL and RFEP student groups were used to assess change in 
selected pre- and post-ELAP measures. These alternative comparisons were 
analyzed due to state data limitations, which precluded use of a single, most 
preferred analysis. Using only EL students in the pre-ELAP measurement and 
both EL and RFEP students in the post-ELAP measurement, a positive and 
statistically significant relationship was found between ELAP and SAT-9 test 
results across all subjects. The size of this relationship is relatively small, with 
average yearly gains in reading and math of approximately 1.4 points, and a gain 
in language arts of 0.8 points. For the approach comparing ELs only in both pre- 
and post-ELAP measurements, ELAP program participation also shows a modest 
and statistically significant gain in SAT-9 reading and math scores. However, the 
approach using ELs and RFEP students in both pre- and post-ELAP 
measurements shows no significant relationship between ELAP and SAT-9 
results. 

• Although a causal relationship can not be claimed (i.e., that ELAP funding is the 
cause of observed differences in test scores), the results do suggest a relationship 
that is positive and statistically significant. 

• Another outcome variable of interest, redesignation, is a particularly difficult area 
to explore given the varying criteria for redesignation used throughout the state 
and the many other factors affecting these rates. While more ELs are redesignated 
in ELAP schools versus non-ELAP schools, as expected given the broad program 
participation, the rates of redesignation are higher in non-ELAP schools. 

This chapter primarily focuses on Research Question 5: 

How have the implementation of Proposition 227 and ELAP provisions affected the 
academic achievement of EL students, as measured by STAR results, redesignation 
rates, drop-out rates, high school graduation exam passing rates, and high school 
graduation rates? 

Possible relationships between ELAP implementation and the academic performance of 
English learners, as measured by STAR results are examined. It also explores possible 
relationships between ELAP and EL redesignation rates. In presenting our analyses, we 
try to point out their limitations, and generally caution against inferring direct causal 
relationships. 
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The first section of this chapter provides a detailed descriptive analysis of the academic 
achievement (based on test scores from the Stanford Achievement Test, 9 th Edition, or 
SAT-9) of cohorts of students in schools that have and have not received ELAP funding. 
It also compares results for English learners (ELs) and English only (EO) students. This 
first analysis does not include information regarding possible demographic or 
socioeconomic differences between the schools that did and did not receive ELAP funds. 
These school-level demographic and socioeconomic differences, such as poverty level, 
parental education, and percent Spanish speakers, are examined in the second section. 

The third section explores possible relationships between ELAP and EL academic 
performance while accounting for, or “controlling for,” the demographic and 
socioeconomic differences across ELAP and non-ELAP schools. It is possible to account 
for, or “hold constant,” such factors as variations in poverty while examining varying 
student outcomes through the use of multivariate regression analysis. The fourth section 
presents descriptive analyses showing changes in EL redesignation rates over the last 
decade, a period that includes the passage of Proposition 227, ELAP implementation, and 
introduction of the California English Language Development Test (CELDT) for use 
statewide. 

Descriptive Analysis of Academic Achievement, by Cohort 

It is difficult to analyze possible relationships between ELAP and EL academic 
performance without being able to match test scores to specific students. Because the 
statewide data do not contain individual student identifiers, the next best scenario is to 
track cohorts of students across the grade span eligible for ELAP funding. This approach 
requires pre- and post-policy data points. We used 1998-99 and 2001-02, respectively. 
Based on these data points, four cohorts of students were analyzed statewide, including 
students in these grades: 

• Second grade in 1998-99 and fifth grade in 2001-02 (Cohort 2-5) 

• Third grade in 1998-99 and sixth grade in 2001-02 (Cohort 3-6) 

• Fourth grade in 1998-99 and seventh grade in 2001-02 (Cohort 4-7) 

• Fifth grade in 1998-99 and eighth grade in 2001-02 (Cohort 5-8) 

These analyses use the Standardized Testing and Reporting (STAR) database, which has 
scaled scores for the Stanford Achievement Test (SAT-9) for reading, math, and language 
arts, to examine changes in academic achievement from the 1998-99 to the 2001-02 
school year. STAR data on poverty level, parental education, language proficiency, and 
primary language are also included in these analyses. These analyses are limited to SAT- 
9 scores for the pre- and post-ELAP implementation years (1998-99 and 2001-02). The 
CAT-6 replaced the SAT-9 in 2003, but the analysis is not extended to this final year of 
available data because there is no good way to compare scores across the two tests. 
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First, a simple descriptive analysis of SAT-9 scores is shown over time (Exhibit 4-1). 
Results are shown for the schools eligible for ELAP that did, and did not, receive these 
funds. It includes 3,731 ELAP-eligible schools, of which 1,076 schools did not receive 
ELAP funds and 2,655 did. 



Exhibit 4-1 : Reading, SAT-9 Mean Scaled Scores by Cohort 1 
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Exhibit reads: Using Approach 1 , the average change in performance for ELs for the period 1 998-02 
in SAT-9 reading scores for Cohort 2-5 was 76 points in schools receiving ELAP funds and 89 for ELs 
in schools not receiving ELAP funds. 



These analyses follow four cohorts of students over time. Ideally, we would follow 
individual students and observe their learning path longitudinally. However, the lack of 
individual student identifiers in state data prevents this, making cohorts the most detailed 
option available. Using school-level data to gauge EL academic improvement on the 
SAT-9 over time raises the issue of how to best categorize students who have been 
redesignated as fluent in English (RFEP). Ideally, we would limit each cohort in the 
initial year of analysis to ELs, and retain all these students in the final year, whether 
redesignated or not. This would allow us to start with a base of just ELs (i.e., those 
potentially benefiting from ELAP), and allow their retention throughout the analysis. 
Unfortunately, the current state data do not allow distinction between previously and 
newly redesignated students. As a result, three different approaches are used, all 
somewhat imperfect, to try to obtain a better understanding of a possible relationship 
between ELAP support and test scores. 



1 The numbers in the “change” column may not match the pie- and post-ELAP score columns due to 
rounding. 
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Approach 1 uses data for ELs for the 1998-99 pre-ELAP score measurement, and data for 
both ELs and RFEPs combined for the 2001-02 post-ELAP measurement. Including 
RFEPs in the post-ELAP measurement captures any ELs that have been redesignated 
since the initial 1998-99 year. However, this approach has the limitation of including 
RFEPs who were already redesignated prior to the ELAP measure (i.e., students unlikely 
to have benefited from ELAP). It is difficult to know a priori the effect the inclusion of 
these students might have on the estimation. Although some RFEPs are included in the 
post-ELAP measure who ideally would be excluded, this is true for both sides of the 
comparison (i.e., the ELAP and non-ELAP schools alike). 

Approach 2 only includes ELs in both the pre- and post-ELAP measurement. While all 
these students could have benefited from ELAP, this approach has the limitation of 
"skimming off" the students who performed well enough to be redesignated, biasing 
average results for the remaining ELs downward. Again, however, this bias applies to 
ELAP and non-ELAP schools. 

Approach 3 includes RFEPs in both the pre- and post-ELAP measures. While this 
approach does not inappropriately remove new RFEPs, it has the disadvantage of 
including students that were RFEPs prior to the pre-ELAP measure, and therefore who 
would not have directly benefited from ELAP. 

Exhibit 4-1 shows the average SAT-9 reading scaled scores and the change in 
performance for the three different approaches; it also shows the performance and change 
in performance for EOs. This information is presented for the four cohorts, by school 
type. 

There are four observations that can be made based on the information in this exhibit: 

• EOs show a higher level of performance in comparison to ELs and ELs/RFEPs 
across all cohorts. 

• The score gains (i.e., the changes in performance) are generally higher for ELs 
and EL/RFEPs than EOs. 

• ELs and EL/RFEPs in non-ELAP schools show lower initial scores than in ELAP 
schools in 1998-99 and yet gain more than those in ELAP schools. This pattern is 
consistent across the three different approaches, and is not uniformly followed by 
the EOs in these schools. 

• Approach 1 shows a greater gain in performance for EL students in non-ELAP 
schools than in ELAP schools in comparison to Approaches 2 and 3. 

The first two observations are not surprising. ELs would be expected to be at a relative 
disadvantage to EOs in English-language reading. As they became more proficient in 
English over time, one might expect them to begin to catch up with their EO 
counterparts, thereby showing relatively greater gains. 
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The last two observations are less intuitive. However, it is important to note that this 
analysis does not control for possible demographic and socioeconomic differences 
between the different types of schools. To allow a more thorough exploration of a 
possible relationship between ELAP participation and EL student performance, it is 
important to include variables such as the percentage of students in poverty and the 
percentage of ELs in the school in the analysis. The multivariate regression techniques 
shown in the third section of this chapter allow for these more complex comparisons. 

The descriptive analyses shown above for reading were also performed for SAT-9 math 
and language arts scores. The language arts results are similar to those presented above 
for reading, and are included in Appendix B. While the math results might be expected to 
differ from reading and language (and are therefore shown below), in fact they also 
mirror the general patterns discussed above for the SAT-9 reading scores (Exhibit 4-2). 
ELs overall showed a greater average change over time than EOs in the same time period. 
Cohorts 2-5 and 3-6 showed the greatest improvement overall, for both ELs and EOs. 

ELs in the non-ELAP schools uniformly showed somewhat greater gains than in the 
schools that received ELAP funds. 



Exhibit 4-2: Math, SAT-9 Mean Scaled Scores by Cohort 2 
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Exhibit reads: Using Approach 1 , the average change in performance for ELs for the period 1998-02 
in SAT-9 math scores for Cohort 2-5 was 85 points in schools receiving ELAP funds and 97 for ELs in 
schools not receiving ELAP funds. 



2 The numbers in the “change” column may not match the pie- and post-ELAP score columns due to 
rounding. 
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Socioeconomic Characteristics of ELAP and Non-ELAP Funded Schools 

This section explores in greater detail the characteristics of schools participating and not 
participating in the ELAP program in the 2001-02 school year. Exhibit 4-3 displays the 
average percentage of students eligible for the National School Lunch Program (NSLP) at 
the school level, which was used in the analysis as the best available measure of school 
poverty. For the first three cohorts, schools that received ELAP showed a higher poverty 
level than schools that did not receive ELAP. For Cohort 5-8, schools receiving ELAP 
showed a similar poverty level to schools that did not receive ELAP. Across all cohorts, 
schools receiving ELAP funding had an average poverty level of 38.4 percent, compared 
to 30.0 percent in the non-ELAP schools. 



Exhibit 4-3: Average Percentage of Students Eligible for Free Lunch, by Cohort 
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Exhibit reads: In non-ELAP schools, 28.3 percent of students in Cohort 2-5 were eligible for free 
lunch; in ELAP schools, 37.7 percent of students in Cohort 2-5 were eligible for free lunch. 
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To further explore the poverty levels of schools receiving ELAP funds, these schools 
were divided into four equal-size groups by the amount of ELAP funds received between 
1999-2000 and 2001-02 (Exhibit 4-4). For example, the first quartile of schools received 
up to $5,623 through 2002, while the fourth quartile received more than $55,121 during 
this same time period. As shown, the difference in average poverty level between the four 
groups receiving ELAP is quite large. For schools receiving the least ELAP funding, the 
average poverty level is 22.8 percent, while for schools receiving the largest ELAP 
allocations, the average poverty level is 59.0 percent. 



Exhibit 4-4: Average Percentage of Students Eligible for Free Lunch by Amount of ELAP 
Funds (Quartiles), 2001-02 
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Exhibit reads: Ranking schools in order by the total amount of ELAP received, schools receiving 
up to $5,623 in ELAP funds had an average of 22.8 percent of students eligible for free lunch. 
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$0 < ELAP funds <= $5,623 < ELAP funds <= $19,525 < ELAP funds Total ELAP funds > 

$5,623 $19,525 <=$55,121 $55,121 
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Exhibit 4-5 shows the relationship between poverty and percentage of English learners. 
ELAP-eligible schools were ranked by poverty level and divided into four groups of 
equal size. The percentage of ELs was compared for schools that did and did not receive 
ELAP funds. The ratio of non-recipient to recipient schools grew with increased poverty, 
with the first quartile showing 330 schools that did not receive ELAP funds as compared 
to 602 schools that did, while the highest poverty quartile had 202 non-ELAP schools and 
731 ELAP schools. As shown, a positive relationship between the percentage of students 
in poverty and the percentage of EL students at the school is observed for both categories 
of schools. 



Exhibit 4-5: Average Percentage of ELs by Poverty Level (Quartiles), 2001-02 




2 . 9 % 18 . 5 % 43 . 2 % 80 . 3 % 

Percent Poverty Level 



□ Average percent of ELs in non-ELAP schools BAverage percent of ELs in ELAP schools 



Exhibit reads: Ranking ELAP-eligible schools by poverty level, non-ELAP schools in the first 
quartile had an average enrollment of 1 .7 percent ELs, while ELAP schools had an average of 4.2 
percent. 



The difference in the percentage of ELs is quite distinct within each school grouping, 
showing the greatest variation in the school group with the highest average poverty level. 
For the 25 percent of schools with the highest average poverty level (80.3 percent), the 
average percentage of ELs in schools receiving ELAP funds was 37.7 percent, while 
schools not receiving ELAP had, on average, 10.5 percent ELs. Perhaps the most 
noteworthy aspect of Exhibit 4-5 is that the average percentage of ELs in schools that did 
not receive ELAP funds remains below 5 percent for about 80 percent of these schools 
(874 schools out of 1,076 non-ELAP schools). For the remaining schools that did not 
receive ELAP (i.e., 20 percent of the sample of non-ELAP schools — 202 schools), the 
average percentage of ELs in the school rises to 10 percent. This is in stark contrast to the 
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schools receiving ELAP, where 46 percent of schools show at least a 10 percent EL 
population (i.e., the first two quartiles of ELAP schools — 1,234 schools). 

Another interesting aspect of Exhibit 4-5 is the percentages of EL students at schools 
receiving ELAP. Half of the schools receiving ELAP had an EL population of 10 percent 
or less, with almost one-fourth of the schools showing an average EL population of 4.2 
percent. This shows that a fairly large number of schools participating in the ELAP 
program have a relatively small EL population. 

Turning to a more detailed descriptive analysis of student achievement, ELAP schools 
were divided into four groups of equal size based on a combination of their poverty level 
and percentage of ELs (Exhibit 4-6). The quartiles range from low-poverty /low 
percentage EL to high-poverty/high percentage EL. 3 

One might anticipate that the pattern would be highest achievement in low-poverty /low 
percentage EL schools and lowest achievement in high-poverty/high percentage EL 
schools. Indeed, this is the general trend. However, looking closely at high-poverty 
schools, Exhibit 4-6 shows that, for SAT-9 reading scaled scores, high-poverty /low 
percentage EL schools have an average score that is just 1 point higher than high- 
poverty /high percentage EL schools. 



Exhibit 4-6: Comparison of ELs’ Average SAT-9 Reading Scores by Poverty Level and 
Percentage of ELs (Quartiles), 2001-02 





Low Percentage of ELs 


High Percentage of ELs 


Low Poverty 


648 


643 


High Poverty 


630 


629 



Exhibit reads: For students attending schools with a low poverty 
level and low percentage of ELs, 648 was the average SAT-9 
reading score. 



Exhibits 4-7 and 4-8 provide results for SAT-9 math and language arts, respectively. 
Here, high-poverty/high percentage EL schools have a higher average score than high- 
poverty /low percentage EL schools by 2 points in math, and by 1 point in language arts. 
While this is a slight reverse from what is observed in Exhibit 4-6, the difference in 
average scores between high-poverty schools with varying percentages of ELs is too 
small to be significant. In the aggregate, these results show the importance of poverty in 
interpreting results, and show that the percentage of ELs may not be significant in high- 



3 Low-poverty /high percentage EL schools are schools that have, on average, 24 percent of students eligible 
for the National School Lunch Program (NSLP) and an average of 28 percent of ELs among their students. 
High-poverty/low percentage EL are schools that have, on average, 49 percent of students eligible for 
NSLP and only 1 percent of English learners in the school. 
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poverty settings. Poverty and percentage of ELs at the school are included as control 
variables in the regression analysis. 



Exhibit 4-7: Comparison of ELs’ Average SAT-9 Math Scores by Poverty Level and 
Percentage of ELs (Quartiles), 2001-02 



Low Percentage of ELs High Percentage of ELs 



Low Poverty 660 656 

High Poverty 637 639 



Exhibit reads: For students attending schools with a low poverty 
level and low percentage of ELs, 660 was the average SAT-9 
math score. 



Exhibit 4-8: Comparison of ELs’ Average SAT-9 Language Arts Scores by Poverty Level 
and Percentage of ELs (Quartiles), 2001-02 





Low Percentage of ELs 


High Percentage of ELs 


Low Poverty 


645 


639 


High Poverty 


624 


625 



Exhibit reads: For students attending schools with a low poverty 
level and low percentage of ELs, 645 was the average SAT-9 
language arts score. 



Another important control variable used in the regression analyses is difference in the 
average education level of the parents of ELAP recipients, as well as the average 
percentage of Spanish speakers among the English learner populations. For schools 
receiving ELAP funding, an average of 43.9 percent of parents have a college education 
or higher. For schools not receiving ELAP, over half (52.3 percent) of parents have a 
college education or higher. 

Another control variable is the average percentage of Spanish speakers in schools 
receiving and not receiving ELAP funds. A simple analysis shows that this variable does 
not vary much, with ELAP schools showing an average of 62.6 percent Spanish speakers 
among their EL population, compared to 66.6 percent at non-ELAP schools. However, it 
is an important control variable because of its relationship to other control variables, such 
as percentage of students in poverty. 
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The next section explores possible relationships between ELAP and the academic 
performance of English learners, while controlling for the important demographic and 
socioeconomic differences discussed above. 

Results of the Multivariate Regression Analyses 

In exploring the relationship between ELAP and academic performance, this section uses 
a theoretical approach with six important characteristics, outlined below. 

Cohorts of students 

This analysis uses the same cohorts of students as the descriptive analysis above. Ideally, 
we would follow individual students to observe their learning paths over time. However, 
the lack of individual student identifiers in the state data prevents this, making cohorts the 
most detailed option available. 

Change over time 

Because we are interested in changes in the academic achievement of English learners, 
we cannot simply compare students’ test scores or average school test scores in a 
particular year. We must use multiple years of comparable data, which allow statements 
about change over time. 

Pre- and Post-ELAP data points 

It is important that the data begin prior to the time of ELAP implementation and extend 
well beyond this point in time to capture the program’s impact. To accomplish this, we 
use test scores from the school years 1998-1999 as pre-ELAP measures and 2001-2002 as 
post-ELAP measures. 

Control groups 

The control group is the group of students or schools for whom the analyzed policy was 
not implemented. This group is critical because it provides a basis for comparison; it 
shows what happens to test scores where the policy is not implemented. For this analysis, 
ELs in schools receiving ELAP funding for different lengths of time and ELs in schools 
not receiving ELAP funds serve as control groups; English only (EO) students in these 
schools serve as a control group as well. 

ELAP as a funding source rather than a specific program 

Because ELAP is a funding source rather than a narrowly defined statewide program, 
schools have discretionary power to decide how to use ELAP funds in order to reach the 
desired goal. This allows ELAP to be implemented differently across participating 
schools. Therefore, this is not constitute an evaluation of a single policy but rather of a 
range of ELAP policies implemented differently across schools. 
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Amount of exposure to ELAP 

ELAP has been in place for varying amounts of time in different schools. Some schools 
received their first notification of apportionment of funds 4 in May 2000, others in March 
2001, and the rest in October 2001. This implies that schools participating in ELAP vary 
in the amount of time they have had to use these funds to provide programs of potential 
benefit to ELs. Because of this, the analyses below include information about the number 
of months since each school received their first notification of fund apportionment. 

To measure the degree of exposure to ELAP, this analysis uses the total number of 
months since the school received ELAP funds 5 for the first time until the end of the 2001- 
02 school year (i.e., August 2002). For example, if a school received ELAP funds for the 
first time in March 2000 and for the second time in October 2001, this school would 
show 29 months of exposure. If another school started receiving ELAP funds in March 
2001, that school would show 17 months of exposure. 

The Regression Model 

This section uses a regression framework to estimate the relationship between ELAP and 
the academic performance of ELs. The dependent variable is the change in the average 
test scores (SAT-9) of EL students between 1998 and 2002. As explained in the prior 
descriptive analysis section, given the absence of individual student identifiers, this 
analysis follows the same cohort of students over time, which raises the issue of how to 
categorize students who have been redesignated as fluent in English. This section uses the 
same three approaches as in the descriptive analysis: 

• Approach 1: EL students in 1998-99 and EL/RFEPs in 2001-02 

• Approach 2: EL students in 1998-99 and EL students in 2001-02 

• Approach 3: EL/RFEPs in 1998-99 and EL/RFEPs in 2001-02 

The objective is to determine whether the length of exposure to ELAP shows a 
statistically significant relationship to the change in the average SAT-9 test scores 
between 1998 and 2002. In other words, we would like to measure the degree of 
correlation between exposure to the program and the observed change in test scores, and 
at the same time to measure if this correlation is statistically significant. We can write this 
question in statistical terms as follows: 

^2002 j ~ ^1998 j ~ a 0 a \ DE j + £j (1) 



4 Districts receive funding approximately four weeks after the apportionment letter. 

5 The intent of the regression analyses contained in this report was to use the date of the first apportionment 
letter as a proxy for the beginning of ELAP fund availability. While final edits received at the time of 
releasing this report clarifies that the date used for the first year is actually based on the date of the grant 
award letter rather than the first apportionment letter, sensitivity analyses show no significant impact on the 
reported findings resulting from this change. 



ENGLISH LANGUAGE ACQUISITION PROGRAM EVALUATION REPORT IV-12 




where 72002 / and Yn mj represent the average test score of student cohort j in a particular 
school in 2002 and 1998, respectively. For example, the average SAT-9 test scores of 
cohort 3-6 (i.e., students who were in third grade in 1998-99 and sixth grade in 2001-02) 
in a particular school would be represented as follows: 7 2 oo2 would be the average test 
score of sixth graders in 2001-02 and would be the average test score of third 
graders in 1998-99. DEj represents the degree of exposure to ELAP of cohort j in a 
particular school. This variable takes the values 0, 10, 17 or 29. ao and 04 are regression 
coefficients and s 7 is an error term. 

The research question is whether the variable DE has had a positive effect on the change 
in the average test score (7 2 oo2 - Yi ( m) of these cohorts of students. If this is the case, the 
coefficient ai will be positive and statistically different from zero. So, we can estimate 
the value of ai in order to answer the research question. If the obtained ai is positive and 
statistically different from zero we can conclude that ELAP shows a statistically 
significant relationship with improvement in the academic performance of English 
learners in California. Note that Equation 1 can be re-written, transferring 7 1993 to the 
right hand side (see Equation 2). Our coefficient of interest is still ai. 

^ 2002 j ~ a Q oi\DE j + a 2 ^ 998 j + £j (2) 

It is important to control for observable differences between schools that did and did not 
receive ELAP funds. This enables us to measure the possible relationship between ELAP 
and academic improvement while controlling for important demographic and 
socioeconomic differences between the ELAP and non-ELAP schools. The control 
variables included in these analyses are the average percentage of students eligible for the 
National School Lunch Program (NSLP) at the school level (as a proxy for the poverty 
level in the school), the interaction between the average percentage of English learners 
and the poverty level at the school, this interaction variable squared, the percentage of 
students with parents with high educational attainment (i.e., with a college education or 
more), the percentage of female students at the school, the percentage of Spanish 
speakers, and a variable that indicates whether the school participates in the English 
Language Intensive Literacy Program (ELILP). 6 Including these control variables, the 
regression to estimate is the following: 

^2002 j ~ + a \ DEj + CL 2 ^[998 y + PX + & j (3) 

where X represents all control variables included in the regression. 

Approach 1 

The first analysis approach uses EL data for the 1998-99 pre-ELAP score measurement, 
and data for both ELs and RFEPs combined for the 2001-02 post-ELAP measurement. In 
this way we can capture any ELs that have been redesignated since the initial 1998-99 
year. However, one of the limitations of this approach is that some of the RFEPs included 



6 The English Language Intensive Literacy Program gives a grant to operate English language/literacy 
programs. Districts received $400 per EL student in K-12, or $3.33 per instruction hour. 
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in the 2001-02 measure were already RFEPs in 1998-99 (and therefore could not have 
benefited from ELAP). 

In order to quantify the degree of influence of this limitation, we analyzed the number of 
RFEP students at the pre and post-ELAP measurement point. Exhibit 4-9 shows the 
number of RFEPs in 1998-99 and 2001-02 by cohort, and shows the percentage of 
“initial” RFEPs, which is the number of RFEPs in 1998-99 divided by the number of 
RFEPs in 2001-02, by cohort. This percentage gives us an estimation of the number of 
RFEPs included in our analysis in the post-ELAP measurement point who were RFEPs in 
1998-99 as well. 

Exhibit 4-9. Number of RFEPs (“Redesignated as Fluent English Proficient”) for the School 
Year 1998-02 and 2001-02, by Cohorts 



Cohorts 


1998-99 


2001-02 


Percentage “Initial” 
RFEPs 


Cohort 2-5 


3,471 


34,687 


10.0% 


Cohort 3-6 


6,845 


39,813 


17.2% 


Cohort 4-7 


12,192 


42,848 


28.5% 


Cohort 5-8 


18,698 


44,184 


42.3% 



Exhibit reads: The number of RFEPs in 3 rd grade in 1998-02 is 6,845 and the 
number of RFEPs in 6 th grade in 2001-02 is 39,813. 



As can be observed in the exhibit above, there are 3,471 students in second grade 
classified as RFEP in 1998-99 and 34,687 students classified as RFEP in sixth grade in 
2001-02. For the purpose of simplification, we can assume that 31,216 (i.e., 34,687- 
3,471) are EL students that have been redesignated as RFEPs in this time period. In other 
words, only 3,471 students (10 percent of the number of RFEPs in 2001-02) in this cohort 
were “initial” RFEPs in 1998-99. For upper grades such as cohort 5-8, the number of 
“initial” RFEPs is much larger, where 42.3 percent of the RFEPs in 2001-02 could have 
been RFEPs in 1998-99. 

With this limitation in mind, Exhibit 4-10 presents the results of the estimation of 
Equation 3 for SAT-9 reading, math, and language arts test scores. The estimated 
regression coefficient ai, which is the measure of the relationship between exposure to 
ELAP program and average SAT-9 test scores, is positive and is statistically significant at 
1 percent, 5 percent, and 10 percent for reading, math, and language arts, respectively. 

The positive sign of the coefficient implies a positive relationship between longer 
exposure to the ELAP program and increased SAT-9 reading, math, and language arts 
test scores for English learners. 
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Exhibit 4-10: Effects of ELAP Funds on SAT-9 Reading, Math, and Language Arts Average 
Test Scores, Approach 1 



Category 


Variable 


SAT-9 
Reading 
Test Score 


SAT-9 
Math Test 
Score 


SAT-9 

Language Arts 
Test Score 


Pre-ELAP test 
scores measure 


Average 1 998-99 SAT-9 ELs test scores 


0.31*** 


0.40*** 


0.39*** 


Exposure to 
ELAP program 


Total Number of Months Since First 
Notified of ELAP Fund Apportionment 


0.12*** 


0.12** 


0.07* 




Percentage students eligible for free lunch 
(poverty level) 


-12.41*** 


-23.14*** 


-15.50*** 




Interaction between poverty level and 
percentage ELs 


-12.77 


9.51 


-2.52 


Control 

Variables 


Interaction between poverty level and 
percentage ELs, squared 


27.03*** 


10.23 


17.05 




Percentage students with parents with 
high educational attainment 


3.95*** 


3.81* 


3.34** 




Percentage female in the school 


7.83* 


7.06 


17.44*** 




Percentage Spanish speakers in the 
school 


-13.06*** 


-19.26*** 


-12.67*** 




Cohort 2-5 


n/a 


n/a 


n/a 


Cohorts of 


Cohort 3-6 


7.17*** 


9.42*** 


5.25*** 


Students 


Cohort 4-7 


12.18*** 


8.29*** 


7.10*** 




Cohort 5-8 


18.30*** 


4.04** 


2.67* 


Participation in 
ELILP 


District have not participated in ELILP 


n/a 


n/a 


n/a 




District have participated in ELILP 


-0.03 


-1.86** 


-0.68 


Constant 




464*** 


432*** 


413*** 


Observations 




1,556 


1,594 


1,577 


R-squared 




0.7 


0.6 


0.6 



Exhibit reads: An EL student in Cohort 2-5 that has received ELAP funds for one month would 
show an average SAT-9 reading test score 0.12 scaled points greater than an EL student in Cohort 
2-5 that has not participated in ELAP program. 

* significant at 10%; ** significant at 5%; *** significant at 1% 



As mentioned previously, the estimated coefficient oti measures the relationship between 
SAT-9 test scores and one additional month in the ELAP program. Although the 
magnitude of this relationship is relatively small, it is statistically significant while 
controlling for observable differences across schools. The obtained monthly ELAP 
“effect” is equal to 0.12 for reading and math. In other words, for each additional month 
in ELAP, English learners increased their average reading and math test score by 0.12 
points. This translates to a yearly effect of about 1.4 scaled score points. These findings 
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suggest that EL students in schools that were apportioned ELAP funds since March 2000 
have increased their average test score by 3.4 and 3.3 for reading and math, respectively, 
in comparison to EL students in schools not receiving ELAP funds. 

The point estimate of the ELAP effect on language arts is slightly different. This estimate 
suggests that each additional month of this program tends to increase the average 
language art test score by 0.07, which translates into a yearly “effect” of less than 1 
scaled score point, which is fairly small. Although we cannot claim a causal relationship 
from these analyses (i.e., that ELAP funding is clearly the cause of the observed rise in 
test scores), overall, the results from these analyses show a relationship between these 
two variables that is positive and statistically significant. 

Exhibit 4-11 graphs the accumulated relationship between test scores and exposure to 
ELAP programs from March 2000 to October 2001. As the exhibit shows, the 
accumulated increase in the academic performance of English learners in schools that 
were first notified of ELAP fund apportionment in March 2000 is larger than the one 
shown by EL students in schools that entered the program later on, in October of 2001. 
Graphically, this is shown by the kink in the line presented in Exhibit 4-11. 



Exhibit 4-11 : Relationship Between Academic Performance and Exposure to ELAP 
Program, Approach 1 

Schools receiving ELAP Schools receiving ELAP Schools receiving ELAP 

funds starting with Year 3 funds starting with Year 2 funds starting with Year 




Months of I mplementation 

Reading (Significant at 1%) Math (Significant at 5%) 

- - > - - Language (Significant at 10%) 



Exhibit reads: For ELAP recipients, the average SAT-9 reading score increase was more than 
one point after ten months of ELAP funding and approximately 3.5 points after 29 months of 
ELAP funding. 
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The estimated effects obtained for the control variables have the expected positive and 
negative signs. For instance, a higher percentage of students eligible for free lunch tends 
to reduce the average reading, math and language test scores. The magnitude of these 
effects indicates that a 1 percent increase in the percentage of students eligible for free 
lunch at the school reduces the school average reading, math and language arts SAT-9 
test scores by 12.4, 23.1 and 15.5 scaled score points, respectively. On the other hand, a 
larger percentage of students with parents with high educational attainment tends to 
increase the average test scores: a 1 percent increase in the percentage of students with 
parents with a college education or higher increases the average reading, math and 
language arts SAT-9 test score by 4.0, 3.8, and 3.3 scaled score points, respectively. The 
results also indicate that female students tend to have higher reading and language test 
scores than males. A 1 percent increase in the percentage of female students increases the 
average reading and language arts test scores at a school by 7.8 and 17.4 scaled score 
points, respectively. Female and male students show no statistically significant 
differences in their math test scores. Finally, these regression results also indicate a 
negative relationship between the percentage of Spanish speakers in the EL population at 
a school and average test scores. A 1 percent increase in the percentage of Spanish 
speakers at a school is associated with a decrease in average test scores in reading, math 
and language arts of 13.1, 19.3, and 12.7 scaled score points, respectively. 

In addition to these control variables, the regression model includes variables that 
indicate the different cohorts used in the analysis. The objective of this is to control for 
possible differences in cohorts of students, given that we cannot assume that all cohorts 
are the same in terms of their academic skills. 

Approach 2 

The second approach left the 2002 RFEP students’ test scores out of the regression, 
comparing just the students currently designated as EL for the pre- and post- 
implementation analysis. The full results of this second regression model are shown in 
Appendix C-l. Exhibit 4-12 shows the relationship between ELAP and academic 
performance for this approach. Leaving out RFEPs in 2001-02 slightly reduces the 
magnitude of the relationship between ELAP exposure and test scores. ELAP’s effect on 
reading scores decreases from 0.12 to 0.11, and on math from 0.12 to 0.10. The 
relationship between ELAP and language art scores becomes statistically insignificant. It 
is important to keep in mind that the limitation of this approach is that it "skims off" the 
students who performed well enough to be redesignated, biasing the average of the 
remaining ELs downward. This bias applies to both the ELAP and non-ELAP schools. 

The estimated effects obtained for the control variables in this approach also have the 
expected signs and are similar to the results obtained in Approach 1. It is worth noting 
that some effects increased in magnitude, such as the effect of poverty level, the 
interaction between poverty and percentage EL, and the percentage of Spanish speakers 
in the school. 
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Exhibit 4-12: Relationship Between Academic Performance and Exposure to ELAP 
Program, Approach 2 



Schools receiving ELAP funds Schools receiving ELAP funds Schools receiving ELAP funds 
starting with Year 3 starting with Year 2 starting with Year 1 




■♦ — Reading (Significant at 1%) A Math (Significant at 5%) 
► Language (Not Significant) 



Exhibit reads: For ELAP recipients, the average SAT-9 language arts score increase was 
approximately one point after ten months of ELAP funding and approximately 3.5 points after 29 
months of ELAP funding. 



Approach 3 

This third approach includes RFEPs in both the pre- and post-ELAP measurement. By 
continuously including RFEP students we avoid skimming off high-performing ELs as 
they are redesignated over this time period, but we also include students who were never 
exposed to ELAP programs. 

Appendix C-2 shows the full set of regression results. The estimated coefficient ai that 
measures the relationship between SAT-9 test scores and one additional month in the 
ELAP program has the correct positive sign but is not statistically significant for any of 
the SAT-9 subjects. It is possible that the limitations associated with this approach 
interfere with the estimation of this model, preventing observation of any relationship 
between academic performance and exposure to ELAP. 

As mentioned at the beginning of this chapter, it is not possible to claim a causal 
relationship between academic performance and ELAP. In other words, it is not possible 
to say that ELAP funds are clearly the cause of the estimated increases in test scores. But 
these results suggest that there is a positive and statistically significant relationship 
between ELAP and the increase in academic performance of English learners. 



ENGLISH LANGUAGE ACQUISITION PROGRAM EVALUATION REPORT IV- 1 8 



Building on the analysis presented in Approach 1, another modified regression model 
was estimated, controlling for the improvement in the average test score of EO students 
in schools that received and did not receive ELAP funds. The results of this modified 
Approach 1 are presented in Appendix C-3. It is important to control for the improvement 
in the academic performance of EO students in order to isolate the specific improvement 
of English learners from a possible overall school effect. Controlling for the improvement 
of EOs tends to reduce the relationship between ELAP and academic performance. More 
importantly, its effect on mathematics skills becomes statistically insignificant. In other 
words, once we control for the change in test scores of EO students, ELAP only shows a 
statistically significant relationship with ELs’ reading scores. 

There are two possible ways to interpret this result. One is that the effect of ELAP 
services may spill over to the population of EO students at schools participating in this 
program. For example, if ELAP funds are used to bolster professional development for 
the instructional staff at the school, all students might be expected to benefit. Another 
interpretation is that other changes that affect EL performance may be happening at these 
schools. These other factors may be improving the performance of all students at this 
school. Under the first interpretation, enhanced EO performance can be seen as a 
supplemental benefit of ELAP. Under the second interpretation, EO performance can be 
seen as a control variable that mitigates the observed relationship between ELAP and EL 
performance gains. It is likely that both of these explanations affect the observed results 
to some degree. 

Relationship Between ELAP and Redesignation Rates 

This section focuses on redesignation rates, as specified in Research Question 5. The 
possible impact of ELAP funding is examined through analyses of redesignation rates 
from 1992 to 2003. Redesignation rates are compared for ELAP and non-ELAP recipient 
schools. Possible ELAP impact is also evaluated on the basis of the total amount of 
ELAP funds received by a school. 

The exhibits in this section use a vertical line to show four events that may be of 
significance in interpreting outcomes. These include 1) implementation of Proposition 
227 in Fall, 1998; 2) introduction of ELAP funds in Spring, 2000; 3) introduction of 
CELDT annual test results for reclassification decisions in Fall, 2001; and 4) introduction 
of English-Language Arts California Standards Test (ELA CST) results for 
reclassification decisions in Fall, 2002. These lines are drawn through the year in which 
they were first implemented. In each of our subsequent analyses, we identify which of 
these events may be influencing outcomes. 

Exhibit 4-13 shows the average number of English learner students redesignated per 
school in ELAP versus non-ELAP schools. It is not surprising that ELAP schools 
redesignate more students than non-ELAP schools, simply because of the larger number 
of English learner students enrolled in these schools. As stated in Chapter 2, there are, on 
average, 21 times as many EL students in grades 4 through 8 in the average ELAP- 
funded district as compared to the average non-ELAP district (1,148 vs. 55). 
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Exhibit 4-13: Average Number of EL Students Redesignated per School, in ELAP Funded 
Schools Versus Non-ELAP Funded Schools, 1992 - 2003 



Prop. 227 ELAP CELDTELA CST 




Non-ELAP schools — -A— ELAP schools 



Exhibit reads: For non-ELAP schools, an average of approximately four students were 
redesignated each year in 1998-99 and 1999-2000, falling to approximately three students 
redesignated per year in 2000-01 . 



Exhibit 4-13 also shows that the average number of redesignated ELs in ELAP-funded 
schools steadily increased from 1992-93 to 2000-01, dipped slightly from 2000-01 to 
2001-02 (the latter year representing the first year that CELDT could be used in 
redesignation decisions), and then increased again slightly from 2001-02 to 2002-03, 
showing an overall upward trend in the number of students redesignated. The non-ELAP 
funded schools show a much smaller average number of students being redesignated each 
year, with this average remaining fairly constant over the ten-year period. 

When comparing ELAP and non-ELAP schools, it is important to take into consideration 
important differences in demographic and socioeconomic characteristics. ELAP and non- 
ELAP schools have notable differences in socioeconomic status; ELAP schools have 
higher rates of poverty than do non-ELAP schools (see Exhibits 4-3, 4-4, and 4-5). Given 
these differences, we need to be cautious when making simple comparisons of 
redesignation rates across schools. 

Exhibit 4-14 shows the average percentage of ELs redesignated per school in ELAP 
versus non-ELAP schools. This is calculated by dividing the number of EL students 
redesignated in one year per school by the total EL population of the school in the 



7 66 of the schools used in the descriptive analyses were dropped from the redesignation analysis due to 
missing data for redesignation and EL counts over the ten-year time period used (1992-2003). 
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previous year. This exhibit shows that the redesignation rates in non-ELAP schools are 
clearly higher than those in ELAP schools. The overall redesignation rates are very close 
to the redesignation rates in ELAP-funded schools, as the vast majority of the EL 
population is found in these schools. 

Exhibit 4-14: Average Redesignation Rates (Weighted) per School, in ELAP Funded 
Schools Versus Non-ELAP Funded Schools 



Prop. 227 ELAP CELDT ELA CST 




— -A— Non-ELAP schools ELAP schools ♦ Overall (weighted) 



Exhibit reads: For non-ELAP schools, an average of approximately thirteen percent of students 
were redesignated each year in 1998-99, 1999-2000, and 2000-01. 



Redesignation rates in non-ELAP schools show a slight upward trend through the passage 
of Proposition 227 in 1998-99, and then the rates decrease in 2001-02 with the inception 
of the use of the CELDT for redesignation decisions. ELAP-funded schools also show a 
slight upward trend in redesignation rates through the year after Proposition 227 passed. 

In the last year prior to the use of CELDT in redesignation, there is a more pronounced 
increase in rates (from 1999-2000 to 2000-01) in these schools. However, as with the 
non-ELAP schools, there is a slight decrease in redesignation rates in 2001-02. The 
increase in redesignation rates before the implementation of CELDT could be due to a 
strong effort reportedly made by many districts to redesignate as many ELs as possible as 
this became a more visible high-stakes accountability measure post-Proposition 227, 9 and 
in anticipation of the new English language proficiency assessment, which was widely 



These rates are weighted by the following formula: number of ELs in the schools divided by the total 
number of ELs in the state, by each year. 

9 

Linquanti, R. (2001). The redesignation dilemma: Challenges and choices in fostering meaningful 
accountability for English learners. Policy Report 2001-1. Santa Barbara: University Of California 
Linguistic Minority Research Institute. 
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expected to be more difficult for EL students than those previously in use. The drop in 
redesignation rates subsequent to the implementation of CELDT could in turn be due in 
part to the difficulty of the newly introduced test, as well as to the net decrease in ELs 
eligible for redesignation after the prior year’s “spike.” The subsequent introduction of 
the ELA CST as another redesignation criterion may have also provided additional 
pressure on redesignation rates in the 2002-03 school year. 

Exhibit 4-15 shows the average redesignation rates of ELAP-funded schools by the 
amount of funds received (where $22,1 15 is the median per year, or cut-off point, to 
differentiate between the two equal-size groups). This exhibit shows that schools 
receiving up to $22,1 15 in ELAP funds have higher redesignation rates than those 
schools that receive more than $22,115. Since schools receive funding solely based on 
the population of EL students, this indicates that schools with smaller populations of ELs 
redesignate higher portions of their students. As previously mentioned, schools with 
higher numbers of EL students and higher rates of poverty redesignate students at lower 
rates. These factors may also create a "ceiling" on the impact of greater ELAP funds on 
increasing redesignation rates within the time frame currently under study. Indeed, this 
may argue for an analysis of EL students’ progress toward meeting redesignation criteria 
as an interim indicator, such as is currently being implemented under the Annual 
Measurable Achievement Objectives (AMAOs) of NCLB Title III. 
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Exhibit 4-15: Average Redesignation Rates (Weighted) per School, by Amount of ELAP 
Funds Received 



Prop. 227 ELAP CELDT ELA CST 




4— Overall School's ELAP funds <= $22,1 15 - -A - School's ELAP funds > $22,1 15 



Exhibit reads: For schools receiving up to $22,1 15 in ELAP funds, an average of approximately 
10 percent of students were redesignated 1998-99, approximately 9 percent in 1999-2000, and 
approximately 1 1 percent in 2000-01 . 



Both groups of schools show slight upward trends in redesignation until the introduction 
of the use of CELDT for redesignation, at which point the rates drop in both groups. 
However, in the following year (2001-02 to 2002-03), the redesignation rates continue up 
again for schools receiving the smaller amounts of funds, whereas the rates in the schools 
receiving greater amounts of funds continue to decrease slightly after the CELDT. It is 
not clear at this time whether the introduction of the ELA CST Basic level as another new 
redesignation criterion may be contributing to either of these effects. The overall 
redesignation rate (i.e., the combined average of the two groups of schools) is very close 
to that of the group receiving over $22,1 15 in funds due to the larger population of ELs in 
these schools. 

Conclusion 

This chapter has explored the relationship between ELAP and the academic performance 
of English learners. The analyses have presented descriptive findings as well as those 
resulting from regression analysis. Regression is designed to isolate the possible impact 
of ELAP on EL educational outcomes from the influence of such other important 
variables as poverty and the percentage of ELs in the school. The results from the 
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regression analysis show the following: Using the ELs (98-99) to EL/RFEPs (01-02) 
analysis approach, a positive and statistically significant relationship was found between 
ELAP and SAT-9 test results. The size of this relationship is relatively small; EL students 
in schools that have received ELAP funds since March 2000 have increased their average 
test score by 3.4 and 3.3 for reading and math, and 1.9 for language arts, respectively, in 
comparison to EL students in schools not receiving ELAP funds. For the ELs (98-99) to 
ELs (01-02) analysis approach, ELAP program participation also shows a modest and 
statistically significant gain in SAT-9 reading scores. However, the EL/RFEPs (98-99) to 
EL/RFEPs (01-02) analysis approach shows no significant relationship between ELAP 
and SAT-9 results. 

Finally, ELAP-funded schools were found to redesignate greater numbers of EL students 
on average, but displayed lower average redesignation rates, when compared to non- 
ELAP funded schools. Differences in redesignation rates by amount of ELAP funding 
appeared to be confounded with concentration of EL students and higher student poverty 
levels. Redesignation rates also appeared to be negatively influenced by the introduction 
of CELDT as a redesignation criterion, although causality cannot be confirmed. 
Redesignation rates may also be influenced by the introduction of ELA CST as a 
redesignation criterion. 
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Chapter 5. Summary and Recommendations 

Highlights: 

• District self-evaluation is not realistic, as attempting to evaluate the impact of an 
individual funding stream is a complex undertaking. 

• As an alternative, the state should consider possible collaboration with selected 
large districts to enable case study evaluations of ELAP. 

• Some additional useful data regarding the implementation of ELAP could be 
collected for all districts. However, increased capacity of the state and districts to 
administer ELAP may be needed in order to support additional data collection. 

• Use the same test statewide over time to the extent possible, in order to monitor 
the progress of categories of students. Ideally, these longitudinal data would link 
individual students over time. 

• Statewide student outcome data are insufficient to provide clear answers 
regarding the degree of ELAP success, although in this report the research team 
has attempted to address the question of effectiveness to the extent possible given 
these limitations. The analyses in this report generally suggest a small, 
statistically significant relationship between ELAP and EL student academic 
achievement. 

The research team believes these results, although modest, are sufficiently promising to 
warrant program continuation with ongoing monitoring and evaluation. This report 
presents an evaluation of the English Language Acquisition Program (ELAP). The 
evaluation has attempted to assess how ELAP is being implemented, what effect it is 
having, and how it can be improved. It was based on information gathered from several 
sources, including results from a survey conducted by AIR/WestEd and the CDE, 
findings from local evaluation efforts, and an analysis of statewide school-level data. The 
evaluation also drew on data from prior phone interviews, surveys, and case study site 
visits related to the larger Proposition 227 evaluation. In addition, the research team 
conducted analyses of statewide student outcome data in an attempt to measure the 
impact of ELAP on EL achievement, and examined redesignation rates over time. This 
chapter summarizes the information presented in the previous chapters, and offers 
recommendations regarding ways ELAP might be made more effective. 

Analysis Approach 

Because statewide data are insufficient to allow tracking individual student academic 
progress over time, the study relied on the best methodology available for gauging 
possible academic progress over the period of ELAP implementation. Cohorts of students 
by grade level were examined, comparing test scores at pre- and post-ELAP 
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implementation points. Multiple regression analyses were used to estimate the 
relationship between ELAP and academic performance of English learners. 

Because current student-level data do not indicate the year of redesignation for a student, 
the preferred model of cohort tracking was not possible, i.e., to start with a base of ELs 
and retain them over time, whether redesignated or not. Because this could not be done, 
three different approaches were used to create alternative views of a possible ELAP 
effect. The first approach includes ELs only in the 1998-99 pre-ELAP measure, and data 
for both ELs and RFEPs combined for the 2001-02 post-ELAP score. Approach 2 
includes ELs in both pre- and post-ELAP measurements, and the third approach includes 
ELs and RFEPs in both measurements. 

Summary of Findings 

District and student participation 

Although less than half of California school districts participate in ELAP, this includes 98 
percent of EL students in grades 4 through 8 in 2002-03. This overall percentage of ELs 
in participating schools is an increase from 91 percent in 1999-2000, the first year of 
ELAP. Thus, the vast majority of districts with large numbers of ELs participate in this 
program. The number of districts participating has also increased in the time the program 
has been in place: 403 districts received ELAP funds in the 1999-2000 school year, 
compared with 516 in the fourth year (2002-03). $51.8 million was allocated the first 
year, $70 million in the second, and $53.2 million in the two subsequent years. 

In the first four years of the program, approximately 71 to 73 percent of the districts in 
California had at least one EL in grades 4 through 8, and were therefore eligible for 
ELAP funds. As expected, districts with higher numbers of ELs were much more likely 
to apply for ELAP. Districts that receive ELAP funds have, on average, a greater number 
and proportion of ELs in grades 4 through 8, and a higher percentage of students in 
poverty. 

Uses of funds 

ELAP funds were used in diverse ways, including core academic, after-school, and 
Saturday programs, staff professional development, language testing and assessment, and 
newcomer classes. The most common use of funds was for ELD instructional programs. 

ELAP strengths and weaknesses as seen by districts 

When asked to identify strengths of ELAP in the Year 4 survey, most districts 
highlighted the focus ELAP funding places on English Learners. When asked about 
constraints, restriction of funds to grades 4 through 8 was viewed as the biggest concern. 
However, some districts saw this as a strength. The second most common constraint cited 
on the Year 4 survey was uncertainty about whether future funds would be available, 
which may affect investment choices and self-evaluation efforts. 
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District self-evaluation 

Although mandated in AB 1116, districts struggled to perform the required self- 
evaluation of the impact of ELAP. Only about 5 percent of the survey respondents (26 
districts) indicated that they had conducted any kind of formal evaluation of ELAP. In 
addition, only 7 of these 26 evaluations explicitly discussed the role of ELAP, and only 5 
of those reflected a clear effort to specifically evaluate its impact on student achievement. 
The analysis plans for these few districts used measures such as statewide tests to follow 
the academic achievement of ELs or qualitative indicators of student progress. These 
findings reflect the complexities associated with this form of self-evaluation. Attempting 
to isolate the impact of an individual funding stream like ELAP is especially difficult. 
This kind of program evaluation appears beyond the capacity of most school districts, 
especially when examples of evaluative models and methods are not provided. 

Despite a general lack of supporting evidence, more than half the districts reported the 
belief that ELAP funds had a moderate to large impact on English language development, 
academic achievement on state content standards, and EL redesignation. Improvement on 
the California English Language Development Test (CELDT) is the area most directly 
affected by ELAP, according to survey respondents. 

ELAP impact 

ELAP recipient schools are demographically different from non-ELAP recipient schools, 
with higher poverty and counts of ELs. When controlling for these differences, the 
overall results 1 suggest a small but statistically significant increase in reading, math, and 
language arts achievement scores in association with ELAP. Across all cohorts, schools 
that received ELAP have an average poverty level of 38.4 percent, compared to 30.0 
percent in the non-ELAP schools. In addition, for schools receiving the least ELAP 
funding, the average poverty level is 22.8 percent, while for the largest school recipients 
the average poverty level is 59.0 percent. Multivariate regression analysis 2 was used to 
estimate the relationship between ELAP and the academic performance of ELs while 
controlling for demographic and socioeconomic differences between ELAP and non- 
ELAP schools. When ELAP schools are classified in terms of length of ELAP exposure 
to the program (i.e., being exposed for 0, 10, 17, or 29 months), the most positive results 
obtained indicate that the monthly ELAP “effect” is equal to 0.12 scaled scores for 
reading and math. That is, for each additional month in ELAP, ELs appear to increase 
their average reading and math test score by 0.12 points. This translates to a yearly 
“effect” of about 1.4 scale score points. For language arts, each additional month of 
ELAP appears to increase the average test score by 0.07, which translates into a yearly 



Three different approaches were designed to analyze cohorts of EL students over time in order to address 
the issue of redesignation. The first approach uses data for ELs for 1998-99 and data for both ELs and 
RFEPs combined for 2001-02, and shows the most positive results. 

2 To examine student achievement in the context of other variables (socio-economic status, parental 
education, English language fluency, etc.), we used a statistical approach called multivariate regression 
analysis. While linear (or bivariate) regressions look at the relationship between two variables, using a 
multivariate regression allows a more complex analysis. When more than two variables can be compared 
(as is the case with multivariate regression), we gain a more complete picture of the interactions between 
variables. 
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“effect” of less than 1 scale score point, which is quite small. (See Exhibits 4-9 and 4-10.) 
Although we cannot claim a causal relationship from these analyses (i.e., that ELAP 
funding causes the observed rise in test scores), overall, the results from these analyses 
suggest a relationship between these two variables that is positive and statistically 
significant. 

Redesignation rates 

Resignation is a required outcome criterion for this evaluation. However, this is a 
particularly difficult area to explore given the varying criteria for redesignation used 
across districts throughout the state, and the many other factors affecting these rates. For 
example, the most noticeable drop in redesignation rates, found for 2000-01, most likely 
results from factors other than ELAP. These include the CELDT, which was first 
administered the prior fall, and the English-Language Arts Content Standards Test (ELA 
CST), which, along with the CELDT, was mandated by the California State Board of 
Education for use in redesignation decisions. While more ELs are redesignated in ELAP 
schools versus non-ELAP schools, as expected given the greater program participation of 
schools with larger EL populations, the rates of redesignation are higher in non-ELAP 
schools. 

Recommendations 

Recommendations derived from a summative evaluation of a funding stream-based 
program like ELAP center on several basic questions: 

• Is there evidence of success? 

• If yes, do the resulting gains appear to warrant the cost? 

• Should the program be continued? 

• If yes, in what ways can it be made more efficient (i.e., greater gains in 
relation to expended funds)? 

In the case of ELAP, the statewide analyses presented in this report suggest modest 
evidence of success. To answer the last three questions, however, a great deal more 
information is needed than is currently available to allow statewide conclusions. In some 
cases, this information could be fairly easily obtained in future years, and in other cases it 
would require more extensive changes in how the state collects student data. Thus, the 
following recommendations focus on what the state can do in the future to further assess 
the relative cost-effectiveness of investments in programs l ik e ELAP. 

District self-evaluation is not realistic. Attempting to evaluate the impact of an individual 
funding stream is a complex undertaking. This challenged the AIR/WestEd research 
team. It is unrealistic to ask districts to determine the efficacy of a funding stream as a 
basis for deciding if it has been a productive investment for the state. 

Collaborate with selected large districts to enable case study evaluations. Some of the 
limitations associated with statewide data, e.g., the inability to link individual student 
outcome measures over time, do not hold for some large districts with substantial internal 
research capacity. However, these districts did not seem to employ this power in response 
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to the statewide requirement that they self-evaluate ELAP. This is likely because ELAP is 
a fairly small program in relation to other district- wide endeavors and, as the future of 
this program has been uncertain, they may have chosen to focus their evaluative efforts 
on larger, district-controlled interventions. However, since the state is attempting to 
determine whether such a program should be continued, and how these funds might best 
be used statewide, perhaps some form of incentive for evaluative collaboration between 
selected large districts and the state should be considered. Such efforts could result in 
case-study data providing evidence as to whether the program appears to be having an 
impact on student performance, and whether some uses of ELAP funds are more cost- 
effective than others. 

Additional useful data could be collected for all districts. The kinds of analyses presented 
in this report — tracking changes in academic progress of English learner students over 
time in relation to their exposure to ELAP interventions — could be enhanced by more 
detailed information in several areas. Statewide data about the exact amounts of funds 
received by individual schools, when the funds were received, and some detail about how 
they were used would allow more fine-grained analyses regarding the potential effect of 
ELAP resources and possibly, the relative efficiencies of one use over another. To 
support this additional data collection, administrative funds may need to be appropriated 
(currently, no funds are made available at the state or district level for data collection). 

Enhance and retain individual student information from state tests. The state now has 
individual records for all students taking a standardized test (e.g., the CAT-6). If 
individual identifiers were retained on these records it would be possible to track the 
academic progress of individual students over time. In addition to that, adding the year of 
redesignation to individual test records would allow the preferred type of cohort analysis 
that was not possible for this report. Given likely concerns regarding confidentiality, 
these identifiers could be scrambled in a uniform way each year to allow linking over 
time, while protecting student identify. Only with individualized linked outcomes over 
time can the impact of such programs as ELAP be fully assessed on a statewide basis. 

Use the same test statewide over time to the extent possible. Having all California 
students take the same test at the same time in the same grade levels over subsequent 
years has substantially enhanced the ability to track the progress of categories of students 
over time. It also increases the potential to assess statewide programs designed to 
enhance student outcomes. While there are sometimes important reasons to change tests, 
e.g. movement from the SAT-9 to the CAT-6, it should also be kept in mind that these 
changes — especially when the tests are not considered to be equated — create serious 
obstacles to ongoing evaluative efforts of state education progress. 

Need for individual student data linked over time. While the state is slowly moving toward 
the realization of this goal, until it is achieved statewide, evaluative efforts using extant 
data will be limited to the fairly imprecise types of student outcome analyses included in 
this report. While we consider it our obligation to derive as much evaluative meaning as 
possible from statewide data in its current form, there will continue to be important 
limitations in the absence of statewide data on the educational progress of individual 
students linked over time. 
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Conclusion 

ELAP was valued by our survey respondents for targeting funds to the state’s substantial 
population of ELs. Most respondents clearly indicated their belief that the attention raised 
for this at-risk and underserved population, and the supplemental programs ELAP 
generated, have been effective in advancing education outcomes for the state’s ELs. 

At the same time, evidence substantiating their belief in the efficacy of this program has 
generally been lacking. The requirement for meaningful district self-evaluation has not 
been met and was unrealistic. Statewide student outcome data are insufficient to provide 
clear answers in regard to the degree of program success. In this report, AIR/WestEd has 
attempted to address the question of ELAP effectiveness to the extent possible, given 
these statewide data limitations. The most optimistic analyses contained in this report 
show a modest, statistically significant relationship between ELAP and EL student 
achievement. Despite our attempts to isolate other important factors likely to influence 
these analyses (e.g., percent EL population in the school and student poverty), current 
statewide data lack the attributes that would be needed to support more definitive 
statements of causality. That is, while we see a statistically positive relationship between 
ELAP and selected student outcome measure, it cannot be said with confidence that 
ELAP has been the cause of these gains. 

We do, however, believe that these analyses are the basis for cautious optimism in regard 
to ELAP. AIR/WestEd believes these findings, although modest, are sufficiently 
promising to warrant program continuation. We also suggest that program 
implementation be enhanced to include some of the recommendations included in this 
report, which would allow better tracking of the extent and ways in which the program is 
impacting the education received by the state's EL population. 
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ABOUT THIS SURVEY 



The English Language Acquisition Program (ELAP) was authorized by California Assembly Bill (AB) 

1 1 16, in 1999, to provide funds to districts to help English learners (ELs) in grades 4 through 8 to 
improve their English proficiency and to better prepare them to meet the state’s academic content and 
performance standards. 

Under AB 1116, any local educational agency (LEA) that receives funding through ELAP must submit a 
report to the Superintendent of Public Instruction on the program’s implementation and effectiveness. 

The CDE will analyze the assessment data for your district to fulfill the program effectiveness component 
of the report. In order to collect and analyze the results of the program implementation, we request that 
your district complete this survey to satisfy this requirement for ELAP evaluation. 

Your responses to this survey will be kept strictly confidential and will be used only for statistical 
purposes. The results will never be presented in any way that would permit any response to be associated 
with a specific school or individual. 



INSTRUCTIONS 

Please answer each question as best as you can. We are interested in your perspective — there are no right 
or wrong answers. When you have finished, Please return it by October 14, 2003 to CDE to the 
following address : 



Attn: ELAP Survey 
Language Policy and Leadership Office 
California Department of Education 
1430 N Street, Suite 4309 
Sacramento, CA 94244-2720 



Thank you for your cooperation in this important effort. If you have any questions or wish further 
information about the survey, you may contact Miguel Navarrette at (916) 319-0269, 
<mnavarre@cde.ca.gov> or Carolyn Macchiavelli at (916) 319-0370, <Cmacchia@cde.ca.gov>. 
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ELAP — LEA Evaluation Report 



In case we have any questions about your survey, please provide the information below: 

Name of LEA: 

County/Di strict Code: / 

Program Contact Person Name: 

Title/Office: 

Phone: ( ) - x 

E-mail address:. 

1. In which of the following school years did your district receive ELAP funding? ( Check all that 



apply.) 


□ 


1999/2000 


□ 


2000/01 


□ 


2001/02 


□ 


2002/03 



2. Does your district allocate ELAP funds to schools? 

□ Yes Go to Question 3 

□ No -> Skip to Question 5 

3. What percentage of your total ELAP budget is used centrally by the district and what percentage is 
allocated to schools? 



To the extent possible, average across all years in which ELAP funds were received ( i. e . , reflecting 
all years for which you have records of your district’s ELAP cdlocations. If you can only answer in 
relation to the most recent year(s) that your district received ELAP funds, please do so.) 

% of ELAP funds are used centrally by the district 

% of ELAP funds are allocated to schools 

Total = 100 % 

4. Which of the following best describes the basis upon which funds are allocated to schools? ( Check 
only one.) 

□ A fixed amount is allocated to schools per EL student 

□ A fixed amount is allocated per school or type of school 

□ Schools receive ELAP funds according to budgets that they submit to the district 

□ Other (please specify ): 
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ELAP — LEA Evaluation Report 



5. Which of the following was supported, at least in part, by ELAP funds? And approximately what 
percentage of your district’s total ELAP budget was allocated to this? 



To the extent possible, average across all years in which ELAP funds were received (i.e., reflecting 
all years for which you have records of your district ’s ELAP allocations. If you can only answer in 
relation to the most recent year(s) that your district received ELAP funds, please do so.) 



a. 

b. 



c. 



d. 



e. 

f. 
g- 



Core academic instructional program 

ELD instructional program 

Extended time program(s) (e.g. after-school, inter-session, 
Saturday school, summer school) 

Newcomer services 

Staff development 

Language testing and assessment 

Other ( please specify ): 



Were ELAP 
funds used for 
this? 

Yes No 


If yes... 
What % of total 
ELAP finds was 
used for this ? 


p 


p 


% 


p 


r 

p 


% 


r 

P 


r 

P 


% 


p 


p 


% 


p 


r 

p 


% 


p 


r 

p 


% 


p 


r 

p 


% 


r 


r 








Total = 100 % 



6. What percentage of total ELAP funds was used for each of the following? 



To the extent possible, average across all years in which ELAP funds were received (i.e., reflecting 
all years for which you have records of your district’s ELAP allocations. If you can only answer in 
relation to the most recent year(s) that your district received ELAP funds, please do so.) 



a. Certificated personnel 

b. Non-certificated personnel.... 

c. Textbooks/materials/supplies 



What % of total 
ELAP funds was 
used for this ? 



% 

% 

% 



d. Technology or equipment 

e. Other (please specify): 



% 

% 



Total = 100 % 
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ELAP — LEA Evaluation Report 



7. To what extent has your district’s ability to use ELAP funds been constrained by the following? 



Large Moderate Small Not at 
extent extent extent all 



a. Restriction of funds to grades 4-8 

b. Uncertainty of available funds 

c. Lack of guidance on how funds can be used 

d. Lack of teachers 



P P P 
P P P 
P P P 
P P P 



e. Lack of classroom space p 

f. Lack of appropriate EL instructional materials p 

g. Delayed receipt of funds p 

h. Other {please specify and elaborate in Question 

9 below)'. p 



P P P 
P P P 
P P P 

P P P 



8. Please describe any other problems your district has encountered in the design and operation of 
ELAP, including identification of any federal, state, or local statute that impedes program 
implementation. Please attach additional pages as necessary. 
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9. Have you used any of the following measures to assess the effectiveness of ELAP in improving 
teaching and learning? 



Yes 



a. California English Language Development Test (CELDT) ... p 

b. SAT-9 p 

c. California Standards Test (e.g., English-Language Arts, 

Mathematics, History-Social Studies, Science) p 

d. District writing proficiency test p 



e. Other district- wide assessment p 

f. Non-cognitive indicators (e.g., attendance, retention) p 

g. Teacher surveys p 

h. Observation of teacher practice p 

i. Other (please specify): ... p 



No 

P 

P 

P 

P 

P 

P 

P 

P 

P 



10. If you answered “yes” to any of the items in Question 9 above, please provide a brief description of 
what you have done in regard to each and what you have found. Please attach additional pages as 
necessary. 
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ELAP — LEA Evaluation Report 



1 1. To what extent has ELAP resulted in each of the following outcomes for English learners (ELs) in 
your district? 







Large 


Moderate 


Small 






extent 


extent 


extent 


a. 


Improved performance on tests assessing ELs’ 
English language development 


.... p 


P 


P 


b. 


Improved performance on tests assessing ELs’ 
grade level achievement on state academic 
content and performance standards 


.... p 


P 


P 


c. 


Increase in school rates of EL redesignation to 
English fluency 


.... p 


P 


P 


d. 


Increase in ELs’ high school completion rates.. 


.... p 


P 


P 


e. 


Other < please specify): 












.... p 


P 


P 



Not No basis for 
at all judgment 

P P 



P P 



P P 

P P 

P P 



12. Do you have evidence that can be used to substantiate your responses to Question 1 1 above? 

□ Yes Go to Question 13 

□ No Skip to Question 14 



13. Please provide a brief narrative presenting this evidence and include examples if possible. Please 
attach additional pages as necessary. 
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ELAP — LEA Evaluation Report 



14. Has your district conducted a formal evaluation of ELAP? 

□ Yes Please Attach and Go to Question 15 

□ No Go to Question 15 



15. Are there any other particular strengths or weaknesses associated with ELAP that are important to 
share? Please attach additional pages as necessary. 
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ELAP — LEA Evaluation Report 



16. Is there anything else that you would like to tell us about your district’s experience with ELAP? 
Please attach additional pages as necessary. 



REMINDER: If your district has conducted a formal evaluation of ELAP, please enclose a copy with 
your completed survey. 



THANK YOU FOR COMPLETING THE SURVEY! 

Please return it by October 14, 2003 to CDE to the following address: 

Attn: ELAP Survey 
Language Policy and Leadership Office 
California Department of Education 
1430 N Street, Suite 4309 
Sacramento, CA 95814 



ENGLISH LANGUAGE ACQUISITION PROGRAM EVALUATION REPORT 



A-9 





APPENDIX B: 

Language Arts SAT-9 Mean 
Scaled Scores, By Cohort 




Language Arts, SAT-9 Mean Scaled Scores by Cohort 





Approach 1 
EL EL/RFEP 


Approach 2 
EL EL 


Approach 3 
EL/RFEP EL/RFEP 


EO EO 


Cohort 


School 

Type 


Pre- 

ELAP 


ELAP Change 


Pre- 

ELAP 


ELAP Change 


Pre- Post- 
ELAP ELAP Change 


Pre- 

ELAP 


Post- 

ELAP 


Change 


2-5 


Non- 

ELAP 


557 


632 


74 


557 


619 


61 


565 


632 


67 


591 


644 


53 




ELAP 


567 


629 


62 


567 


623 


55 


568 


629 


61 


588 


645 


57 


3-6 


Non- 

ELAP 


570 


635 


65 


570 


629 


59 


576 


635 


59 


606 


649 


43 




ELAP 


580 


641 


61 


580 


632 


52 


584 


641 


57 


603 


654 


50 


4-7 


Non- 

ELAP 


584 


642 


57 


584 


629 


44 


592 


642 


50 


619 


657 


38 




ELAP 


598 


644 


46 


598 


637 


39 


604 


644 


39 


622 


659 


36 


5-8 


Non- 

ELAP 


605 


639 


34 


605 


633 


28 


607 


639 


32 


631 


661 


30 




ELAP 


610 


647 


37 


610 


641 


30 


614 


647 


33 


635 


663 


29 
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APPENDIX C: 



Regression Results for 
Student Achievement 
Analysis Models 




Appendix C-1 : Approach 2, Regression Results Using ELs as “Pre” Group and ELs as “Post” Group 



Dependent Variable: Average 2001 Test Score of ELs 





Reading 


Math 


Language 


Average 1998 Test Score of ELs 


0.19 


0.32 


0.27 


ELAP Variable: 


(0.02)*** 


(0.02)*** 


(0.02)*** 


Total Number of Months Since Receiving ELAP Funds 


0.11 


0.10 


0.04 


Control Variables: School Characteristics 


(0.04)*** 


(0.05)** 


(0.04) 


Percent Students Eligible for Free Lunch 


-20.08 


-32.87 


-24.43 




(2.67)*** 


(3.50)*** 


(2.82)*** 


Interaction Variable Between EL and Poverty 


31.84 


61.01 


46.68 




(9.33)*** 


(12.34)*** 


(9.93)*** 


Interaction Variable Between EL and Poverty, Squared 


-8.82 


-30.24 


-22.46 




(11.40) 


(15.14)** 


(12.18)* 


Percent Students with Parents with High Educational Attainment 


0.89 


0.04 


0.05 




(1.63) 


(2.17) 


(1.74) 


Percent Female in the School 


9.55 


6.87 


16.39 




(5.16)* 


(6.87) 


(5.54)*** 


Percent Spanish Speakers in the School 


-16.34 


-23.59 


-16.77 


Control Variables: Student Cohort 

Control Group: Cohort 2-5 (2 nd Grade in 98-99 and 5 th Grade in 01-02) 


(2.34)*** 


(3.10)*** 


(2.49)*** 


Cohort 3-6 (3 rd Grade in 98-99 and 6 th Grade in 01 -02) 


9.86 


9.00 


6.07 




(0.81)*** 


(1.05)*** 


(0.80)*** 


Cohort 4-7 (4 th Grade in 98-99 and 7 lh Grade in 01-02) 


14.12 


8.79 


7.68 




(1.53)*** 


(1.92)*** 


(1.52)*** 


Cohort 5-8 (5 th Grade in 98-99 and 8 lh Grade in 01-02) 


24.89 


9.60 


7.39 


Control Variables: Participation in Other Similar Policies 

Control Group: School Does not Participate in ELILP 


(1.63)*** 


(2.08)*** 


(1.58)*** 


Participates in English Language Intensive Literacy Program (ELILP) 


-0.94 


-3.36 


-1.92 




(0.71) 


(0.94)*** 


(0.75)** 


Constant 


519 


465 


473 




(11.19)*** 


(12.82)*** 


(12.39)*** 


Observations 


1556 


1594 


1578 


R-squared 

Standard errors in parentheses 


0.6 


0.5 


0.4 



* significant at 10%; ** significant at 5%; *** significant at 1% 



ENGLISH LANGUAGE ACQUISITION PROGRAM EVALUATION REPORT C- 1 






Appendix C-2: Approach 3, Regression Results Using ELs as “Pre” Group and ELs + RFEPs as “Post” 
Group 



Dependent Variable: Average 2001 Test Score of ELs + 
RFEPs 





Reading 


Math 


Language 


Average 1998 Test Score of ELs + RFEPs 


0.36 


0.42 


0.41 


ELAP Variable: 


(0.01)*** 


(0.02)*** 


(0.02)*** 


Total Number of Months Since Receiving ELAP Funds 


0.03 


0.06 


0.03 


Control Variables: School Characteristics 


(0.03) 


(0.04) 


(0.03) 


Percent Students Eligible for Free Lunch 


-15.04 


-21.46 


-15.98 




(2.16)*** 


(2.72)*** 


(2.33)*** 


Interaction Variable Between EL and Poverty 


7.95 


14.59 


7.70 




(7.66) 


(9.67) 


(8.27) 


Interaction Variable Between EL and Poverty, Squared 


3.31 


9.72 


6.48 




(8.48) 


(10.72) 


(9.18) 


Percent Students with Parents with FHigh Educational Attainment 


6.12 


5.92 


4.75 




(1.43)*** 


(1.80)*** 


(1.55)*** 


Percent Female in the School 


8.84 


5.77 


12.43 




(4.00)** 


(5.01) 


(4.35)*** 


Percent Spanish Speakers in the School 


-8.94 


-14.82 


-8.53 


Control Variables: Student Cohort 

Control Group: Cohort 2-5 (2 nd Grade in 98-99 and 5 th Grade in 01-02) 


(2.12)*** 


(2.67)*** 


(2.29)*** 


Cohort 3-6 (3 rd Grade in 98-99 and 6 th Grade in 01 -02) 


6.18 


8.16 


4.70 




(0.72)*** 


(0.90)*** 


(0.73)*** 


Cohort 4-7 (4 th Grade in 98-99 and 7 lh Grade in 01-02) 


9.84 


7.08 


6.73 




(1.31)*** 


(1.59)*** 


(1.35)*** 


Cohort 5-8 (5 th Grade in 98-99 and 8 lh Grade in 01-02) 


15.98 


3.54 


2.59 


Control Variables: Participation in Other Similar Policies 

Control Group: School Does not Participate in ELILP 


(1.39)*** 


(1.72)** 


(1.38)* 


Participates in English Language Intensive Literacy Program (ELILP) 


2.06 


0.83 


0.93 




(0.60)*** 


(0.76) 


(0.65) 


Constant 


433 


413 


397 




(8.26)*** 


(9.17)*** 


(9.34)*** 


Observations 


2770 


2840 


2815 


R-squared 


0.6 


0.5 


0.5 



Standard errors in parentheses 

* significant at 10%; ** significant at 5%; *** significant at 1% 
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Appendix C-3: Modified Approach 1, Regression Results Using EOs as Control Variable, ELs as “Pre” 
Group and ELs + RFEPs as “Post” Group 



Dependent Variable: Average 2001 Test Score of ELs + 
RFEPs 





Reading 


Math 


Language 


Average 1998 Test Score of ELs 


0.34 


0.47 


0.43 


ELAP Variable: 


(0.02)*** 


(0.02)*** 


(0.02)*** 


Total Number of Months Since Receiving ELAP Funds 


0.09 


0.07 


0.05 


Control Variables: School Characteristics 


(0.04)*** 


(0.04) 


(0.04) 


Increase in Average Test Score of EOs (1998-2001) 


0.20 


0.44 


0.31 




(0.03)*** 


(0.03)*** 


(0.03)*** 


Percent Students Eligible for Free Lunch 


-11.84 


-16.52 


-12.63 




(2.39)*** 


(2.94)*** 


(2.48)*** 


Interaction Variable Between EL and Poverty 


-16.37 


1.98 


-8.31 




(8.37)* 


(10.30) 


(8.70) 


Interaction Variable Between EL and Poverty, Squared 


29.44 


12.70 


19.37 




(10.24)*** 


(12.67) 


(10.70)* 


Percent Students with Parents with High Educational Attainment 


4.59 


3.26 


3.64 




(1.47)*** 


(1.82)* 


(1.54)** 


Percent Female in the School 


5.42 


-0.95 


9.34 




(4.62) 


(5.74) 


(4.90)* 


Percent Spanish Speakers in the School 


-12.40 


-19.11 


-12.60 




(2.10)*** 


(2.58)*** 


(2.18)*** 


Control Variables: Student Cohort 


Control Group: Cohort 2-5 ("2 nd Grade in 98-99 and 5 m Grade in 01-02) 




10.17 


10.70 


Cohort 3-6 (3 rd Grade in 98-99 and 6 th Grade in 01 -02) 


(0.82)*** 


(0.87)*** 


(0.71)*** 




16.64 


16.48 


11.55 


Cohort 4-7 (4 th Grade in 98-99 and 7 lh Grade in 01-02) 


(1.49)*** 


(1.67)*** 


(1.39)*** 




23.04 


17.30 


8.86 


Cohort 5-8 (5 th Grade in 98-99 and 8 lh Grade in 01-02) 


(1.59)*** 


(1.90)*** 


(1.48)*** 



Control Variables: Participation in Other Similar Policies 


0.30 


-0.67 


0.14 


Control Group: School Does not Participate in ELILP 


(0.63) 


(0.78) 


(0.66) 


Participates in English Language Intensive Literacy Program (ELILP) 


435 


355 


377 


Constant 


1552 


1591 


1573 




0.7 


0.7 


0.6 


Observations 


0.34 


0.47 


0.43 


R-squared 


(0.02)*** 


(0.02)*** 


(0.02)*** 



Standard errors in parentheses 

* significant at 10%; ** significant at 5%; *** significant at 1% 
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